I recall the events of 9/11 as an earlier generation recalled the death of President Kennedy. The difference being that this was not a tragedy I saw on television. Having worked downtown at the time, I was on my way to my office when the two hijacked aircraft hit the Twin Towers and I arrived in time to see one of the towers fall. The grey clouds of ash still float across my memory, interspersed with mental snapshots of people falling to their deaths.

I could never imagine, at the time, that there would be a 9/11 museum. Not only because the events of that day seemed too horrific and surreal to fully digest, much less to recall with the luxury of time and distance, but also because I never expected that those events would be as unique as they have remained. Seeing America under attack on 9/11, I had little doubt that we would witness attacks of equal if not greater magnitude in the years ahead. For weeks afterward I half cringed every time I walked through a crowded public place in Manhattan, knowing that so many people clustered together could be an irresistible target for a suicide bomber.

Mercifully, my worst fears have not come true. To be sure, there have been terrorist attacks on American soil since 9/11–attacks such as those on the Boston marathon and at Fort Hood. There have been even more foiled plots such as the one at the Mohammad cartoon contest in Garland, Texas. But neither Al Qaeda nor any other group has succeeded in pulling off an attack of 9/11-scale.

I thought about that as I walked Sunday through the 9/11 Museum in downtown New York across the street from where I used to work. It was a haunting and moving experience–especially seeing the pictures of all the victims and hearing the recordings of passengers on the doomed airplanes calling their loved ones, telling them not to worry, something has gone slightly wrong but everything will be ok. It was nearly unbearable.

It caused me to reflect that we were monumentally unlucky on 9/11 and we have been monumentally lucky ever since.

But that doesn’t mean we can or should expect our luck to continue indefinitely, especially not if we dismantle the defenses that have kept us safe. That seems to be what a left-right coalition of House and Senate members is trying to achieve by making it harder for the National Security Agency to search telephone records for links between terrorists. The Patriot Act, the cornerstone of homeland security since 9/11, is due to expire on May 31 and these lawmakers are holding its renewal hostage until they get what they want–which is weaken our defenses against terrorism.

Their rationale is that the current system, as exposed by Edward Snowden, trespasses on our liberties even though there is no evidence of the NSA abusing its authority in any way. Contrary to what the fear-mongers would have you believe, the metadata collection does not allow government gumshoes to listen in to your calls to your Aunt Sally; that still requires a court order.

I couldn’t help wishing, as I toured the 9/11 Museum and ground zero, that all of the lawmakers who are blocking passage of the Patriot Act should be required to take the same tour–to remember what it was really like on 9/11 and how easily the deadliest attack ever on American soil could have been prevented if there had been better intelligence collection and law enforcement work beforehand. The post-9/11 reforms have corrected many of the problems that used to exist, but we have become so complacent in the years since that it is all too easy to forget the kind of threat that we faced then–and still face.

The core of Al Qaeda may have been greatly weakened by American actions in Pakistan and Afghanistan but the jihadist threat has since metastasized and in many ways it’s gotten worse than it was on 9/11. Osama bin Laden may be resting in his watery grave but groups from ISIS to Al Qaeda on the Arabian Peninsula have proven themselves to be every bit as fanatical, if not more so, and in many ways they are also proving even more successful and resourceful.

On 9/11 Sunni jihadists controlled most of Afghanistan. They no longer control Afghanistan but they do control a vast caliphate encompassing half of Syria and a third of Iraq. They also control substantial areas of Pakistan and Yemen. Meanwhile their opposite numbers among Shiite jihadists are taking control of much of the rest of Syria, Iraq, and Yemen.

Now is no time to let down our guard. Rather it is time to remember the horrors of 9/11 and to vow that we shall do whatever it takes to avoid another such calamity. And if that means a slight and inconsequential infringement on civil liberties, so be it.

The next time we won’t have the luxury of saying we could not anticipate what was to come. If you want to experience the shadow that looms over our future as well as our past, all you have to do is visit the 9/11 Museum.