My dashboard was finished. It was also empty.

Every stage was approved. Every check was green. I opened Grafana to look at my finished vulnerability dashboard, and it looked great!

It just didn't have any vulnerabilities in it.

Technically, I suppose that's the best possible security posture. Unfortunately, it wasn't the real one.

The original idea was fairly simple. Our CISO wanted a way to see what was happening in our AWS environment without having to learn AWS, which is an entirely reasonable thing to want. The plan was to build a Grafana dashboard where he could log in, look at a few charts, and understand what was going on. No digging through AWS consoles, deciphering acronyms, or waiting for someone to assemble a spreadsheet.

I built it using AIDLC, AWS's methodology for developing software with AI. I ran the process in Kiro. The basic idea behind AIDLC is that the AI does much of the planning and typing, while you make the decisions. It starts with an inception phase, where you work out what you're building, and then moves into construction, where the system is designed and built one piece at a time. Each stage begins with a planning document containing a lot of questions for both functional and non-functional requirements. You answer them, review the resulting plan, and approve it before moving on.

There is a lot of paperwork. The AI never gets tired of asking questions, and unlike a human coworker, it doesn't seem to take “I think we're good here” as a hint that perhaps we have discussed a topic enough.

Anyway, what started as a dashboard became quite a bit more than a dashboard.

Under the two Grafana pages, Kiro built six OpenTofu modules. OpenTofu is the open-source relative of Terraform, and it lets you define cloud infrastructure in code. It also wrote the SQL for sorting through millions of security findings, a daily snapshot job, a small tool for recording who owns what, and 242 automated tests.

I typed very little of the actual code myself. I approved most of it, argued with some of it, and caught a few things it got wrong. This sounds easier than it was.

AIDLC feels a little like taking a very long multiple-choice test where the questions become increasingly technical as you go. Where should the data live? Who should own it? How strict should the guardrails be? The AI usually offered two reasonable options, A and B, and I regularly responded with something like, “Neither, here's what I actually want,” or, “A, but also…” At one point, it asked whether a guardrail should block a broad list of actions or a smaller, more specific list. I proposed a third option: block everything except the few actions the dashboard genuinely needed. I felt quite clever about that answer! I also lost about a week because of answers like it. Before I could propose my own option, I had to understand the original choices, research their implications, and then design something else.

There was one decision I slowed down for deliberately: how long to retain the audit records. That setting has to be chosen when the storage is created, and it can't be changed later. I don't mind fixing a dashboard after deployment, but I didn't want to discover six months later that I'd made the wrong permanent decision because I was eager to get through a planning document.

Eventually, everything was built and the dashboards were ready.

They were also empty.

The problem turned out to be a missing setting. The dashboards were defined in code, and the code omitted something required by the Athena plugin, which is what Grafana uses to talk to the database. Grafana crashed in my browser while trying to build each query, before it sent anything to Athena.

This meant I still had no idea whether the underlying data worked. The dashboard had never actually asked for it.

What bothered me was that every automated check had passed. The connection test passed. The infrastructure validated. The queries worked perfectly from the command line. None of those checks opened the dashboard in a browser, which was the one place where the system was broken.

Opening it in a browser seems obvious now. It did not seem obvious when I was looking at a large collection of green check marks and assuming one of them meant the dashboard worked.

A browser check is on the to-do list.

We (Kiro and I) fixed the missing setting, and the dashboard finally filled up. Then I started looking at what it was actually showing.

About 99 percent of the results weren't vulnerabilities. They were compliance checks, most of which had already been resolved. Each finding also appeared around 84 times, so there were a great many rows and very little useful information. Once I got through all of that, the actual number of open vulnerabilities was 81.

The requirements had said to include all findings flowing through Security Hub. The AI did exactly that. The problem was that nobody, including me, had stopped to ask the more basic question: what counts as a vulnerability?

Apparently, “a failed compliance check” and “a vulnerability” are not interchangeable just because they arrive through the same AWS service. We removed the compliance findings from this dashboard. They'll eventually get a dashboard of their own, which is now another item on the list.

So the dashboard went from empty to full of the wrong things. I suppose that's progress.

The tests created a similar problem. The code passed every one of them, but once we ran it against the real services, the database began rejecting things no test had known to question.

The AI named one stage of a query normalize, which happens to be a word the database won't allow as a name. It requested a table setting that sounded completely plausible but did not exist. It configured an alarm using a time window longer than the service permits. It also deployed a job without including the SQL files the job needed in order to run.

None of these was a particularly difficult fix. They were just difficult to find before the real system objected to them. Afterward, I wondered whether a SQL checker might have caught two of the database problems. Rather than assuming it would, we gave the checker the broken queries and waited to see what happened.

It accepted both of them without complaint.

Had I skipped that experiment, I probably would have added the checker, watched it pass, and felt reassured. Instead, I learned that a checker also needs to prove it can catch the thing you're worried about. This is probably obvious, but a surprising amount of this project consisted of learning obvious things by first doing them the other way.

One part did work exactly as intended.

Early in the project, we established a rule that any fix affecting something outside our own project had to stop for approval. Later, the dashboard connection failed, and one of the possible fixes involved changing a role owned by another team. The AI stopped and presented the options rather than quietly modifying something it didn't own. I chose the fix that stayed within our project. Given how many other assumptions had slipped through, I was relieved to see that this particular boundary held.

I've been trying to decide what I would do differently next time.

The first thing is to spend more time in inception. I began with what I thought was a clear vision: build a vulnerability dashboard for the CISO. As it turns out, a clear vision is not the same thing as a complete plan. Mine didn't answer the most basic question about what counted as a vulnerability. We also changed our minds about which account should own the data, then later changed them back.

I wanted to get through the planning stage so I could start building. In retrospect, more time there might have saved time later. If I had worked through more of the definitions and ownership questions at the beginning, I might not have had to research every decision while construction was already underway.

The second thing is that a second agent helps, but it doesn't remove the need to pay attention. I gave part of the work to a second Claude to review Kiro's output, and that sped things up quite a bit. I still had to read what both agents told me because their summaries didn't always match what the system actually did. Without the second agent, I might still be working through some of this. Without my own review, I would have trusted several things that weren't true.

And, again, any checker has to demonstrate that it can catch the problem before I trust the green check it produces.

The third thing is simply to open the finished thing and use it. Every automated check was green, and the dashboard was still broken. The tests were useful, but they only answered the questions they had been written to answer. None of them answered, “Can a person open this dashboard in a browser and see vulnerabilities?”

There is one question I still haven't resolved.

Whenever AIDLC gave me two options, I tended to study both and then design my own hybrid. That produced decisions I was more comfortable with, but it also added a lot of time. One set of answers alone cost me about a week. The alternative is to accept the default, keep moving, and fix it later if necessary. I don't know whether that would have been better. Some of these decisions were easy to change, while others—like the audit-retention setting—weren't. Perhaps the real answer is to spend more time up front defining which decisions deserve careful research and which ones are safe to revisit later.

Ask me again after the next project.

For now, I'm still working through this one and fixing things as I find them. It isn't done, exactly. But it is a lot less empty than it used to be.