lol I guess there's a fun new feature in the March patch Tuesday update for Windows server. The new feature is a memory leak that can cause Domain Controllers to spontaneously reboot. Affects every supported version of Windows Server. Good thing I haven't updated yet.
lol I guess there's a fun new feature in the March patch Tuesday update for Windows server. The new feature is a memory leak that can cause Domain Controllers to spontaneously reboot. Affects every supported version of Windows Server. Good thing I haven't updated yet.
lol I guess there's a fun new feature in the March patch Tuesday update for Windows server. The new feature is a memory leak that can cause Domain Controllers to spontaneously reboot. Affects every supported version of Windows Server. Good thing I haven't updated yet.
We saw those reports earlier in the week and put a hold on those patches until there's a fix. We have a week delay baked into our patching schedules specifically to avoid getting caught out by DOA patches like this.
Just remember that half the people you meet are below average intelligence.
Yeah I usually push my patches out the week before patch tuesday, give it a full 3-4 weeks before applying. If there's an "oh shit" zero day I may go a week earlier than normal, but the cadence of being 3-4 weeks behind on most patches has served me well.
Yeah I usually push my patches out the week before patch tuesday, give it a full 3-4 weeks before applying. If there's an "oh shit" zero day I may go a week earlier than normal, but the cadence of being 3-4 weeks behind on most patches has served me well.
On domain controllers, because Active Directory handles redundancy so well, I do them spaced out over the two weeks after patch Tuesday.
One of them always gets patched right away. That's the canary in the coal mine.
every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
I was gonna say it's the difference between my po-dunk redneck engineering ass and the very qualified smarty smarts that inhabit this thread.
I'm currently trying to redesign our server room and by that I mean move shit around because I got too fat to be able to get behind the servers because someone decided to only allow a foot of space between the back of the server racks and the wall. But also I gotta do it without spending any money, of course.
Sounds like you should tell management you can fix the server room for $2000 - by letting you buy a gym rack and a Peloton. (Sorry; I'm currently way too close to 200lbs at 46 so now I take the stairs any time I have to go between floors and our industrial area)
I mean, pretty much. While it's entirely my fault that I wasn't able to adjust my daily routine, I was actually at my healthiest when I was still working from home. Being at home helped me stick to a strict diet, I could hit my bike over my lunch break, go for a walk when I needed to take a 10 minute break, etc.
But that aside the only other guy in the department is probably pushing 400, like, I gotta move this shit. It shouldn't be like that in the first place, and if I get hit by a truck and someone needs to access the back he's fucked.
Even ignoring the human element, what if you need to get a piece of equipment back there for a building repair or something?
Edit: now I have mental images of a poor tech guy caught up in cat5 cable in the 1 foot gap behind the servers, found several years after his death and desiccated like an ancient dead rat.
That was written by a dude in 1994 as a temporary solution as Microsoft was porting the UI it was developing for Windows 95 into NT4 until the "good" UI was done. the 32GB FAT32 format limit was an arbitrary number hard coded into the UI because it needed to be a number, 32GB was larger than anything you'd fine in 1994, and would work until a permanent UI was written.
It's still in use in 2024.
Kind of funny as hell that the "I'll just do this thing until I ca.....oh god it's in production now" is also a thing that happened to Microsoft, and not just every Small/Medium Business in the world
To make a long story short, finally migrating the last users and bits off of our on premises Exchange server. Need to set up scan to email on our MFP's through office 365, not the local server. First machine I try, which is the biggest one in the building and closest to my desk, doesn't work. Spend a couple days troubleshooting my setup, open a ticket with Microsoft to make sure my setup is correct, it is, but the machine just won't connect to the office 365 smtp server. Plug my laptop into that network port to make sure that there's not some weird blocking on the switch to keep the printer from talking to the internet, and everything is fine.
After a solid 3 days fighting with it, I try a different machine, and it works the first time.
So I go back to the original machine, after some digging, whoever set this printer up approximately 8 years ago typo'd two things in the network settings, the subnet mask and default gateway. So it could communicate with anything on the last octet range, but nothing else. That worked fine since the machine and the exchange server were on the same subnet for some reason, but it couldn't talk to any other subnet or the internet.
It's been "working fine" for 8 years so nobody really noticed. Kill me.
We still have an internal SMTP server for anonymous auth stuff, like printers. Nothing will accept a message from it unless explicitly told to, so sometimes I have to tell people they'll need to scan to their email first and then forward to the external party.
We set up a... what used to be a connection filter, not sure what it is now, in Exchange Online so it will accept messages from that NAT IP.
While O365 does have an anonymous SMTP setup for that sort of situation we already had this SMTP server in place, so it's only been used for stuff that is off our internal network.
Ok, so the continuing saga of that stupid exchange server. To give a bit of background it is a 2010 server, that wasn't touched for years because the people before me here never updated anything until they absolutley had to and decided that everything was the next guy's problem, and that next guy is me.
That server ran some local distribution groups, and about 30 users that we didn't want to pay for an Exchange online license for (people like warehouse/delivery guys who needed email and phones). But since it was Exchange 2010 and it's been out of support for a couple of years, Microsoft is starting to block it from talking to Exchange Online, hence I was finally able to convince the powers that be that I needed to carve out time to get us off of it.
Because it is 2010, even though I do have everything off of it, the process of getting it off my network is turning into a nightmare, since you do still need on premesis tools to manage exchange online in 2024. There are powershell tools you can use now to do this instead of needing a full (free) Exchange server set up for recipient management, but those only work if your exchange enviornment was on 2013 or later. So... here is what I need to do to decomission Exchange 2010:
1: upgrade the Exchange 2010 server from 2010 SP3 to the latest update which is SP3 update rollup 32. (the patch level of the server was a build from 2013, I need to be on a build from at least late 2015 to perform steps 2 and 3)
2: Install an exchange 2016 instance. This will upgrade the on prem Exchange schema from 2010 to 2016
3: enable co-existance of Exchange 2010 and 2016. (this is why step 1 is necessary, need to be on a a build that supports co-existence)
4: run the hybrid exchange wizard on exchange 2016 so Exchange Online talks to that server instead of the 2010 server.
5. I can then go through the process of decomissionsing the 2010 server.
6 (Optional step). Install the powershell tools that do allow you to manage exchange online users without a full exchange server in the enviornment. This would upgrade the on prem schema to 2019. But I may just keep the 2016 server around for ease of use since we can use gui tools to manage what we need to manage still.
Why, you ask do I not just install Exchange 2019? Well you can't enable coexistence between Exchange 2010 and 2019. You have to first go to 2016.
There's a reason many (most?) companies of any size keep an internal Exchange server just as an SMTP relay. Sometimes it's also used for management.
BTW, wunderbar, all that sounds reasonable and not too big of a deal. Yeah, it sucks shit that they left a 2010 server running for 14 years but doing a double upgrade to 2016 then 2019 shouldn't be a hair-puller.
Feral on
every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
Coincidentally, I was just on a call this morning about virtual desktop infrastructure (VMware Horizon) and somebody asked "do we need to set up XYZ datacenter with HA?" and the unanimous response was "there's always value in HA"
In this case all it meant was:
two load balancers
two Horizon connection servers
three Horizon virtualization hosts
with hyperconverged infra so there's redundant storage
not an enormous investment
Just an example of how HA can refer to simple redundancy in small environments. (I'm not arguing that other interpretations are wrong, just expressing how the phrase can be used.)
Feral on
every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
There's a reason many (most?) companies of any size keep an internal Exchange server just as an SMTP relay. Sometimes it's also used for management.
BTW, wunderbar, all that sounds reasonable and not too big of a deal. Yeah, it sucks shit that they left a 2010 server running for 14 years but doing a double upgrade to 2016 then 2019 shouldn't be a hair-puller.
yeah it's just a pain in the ass, and something I've been begging for the time to do for a year and a half and then we had a cyber security audit done and shockingly a Server 2008 R2 server Running a build of Exchange 2010 from 2013 was found to be a pretty big vulnerability in our network and suddenly leadership was very interested in having it be fixed yesterday.
There's a reason many (most?) companies of any size keep an internal Exchange server just as an SMTP relay. Sometimes it's also used for management.
BTW, wunderbar, all that sounds reasonable and not too big of a deal. Yeah, it sucks shit that they left a 2010 server running for 14 years but doing a double upgrade to 2016 then 2019 shouldn't be a hair-puller.
yeah it's just a pain in the ass, and something I've been begging for the time to do for a year and a half and then we had a cyber security audit done and shockingly a Server 2008 R2 server Running a build of Exchange 2010 from 2013 was found to be a pretty big vulnerability in our network and suddenly leadership was very interested in having it be fixed yesterday.
oof, i know that feeling
besides the stress and frustration of having to do it NOW rather than in a well-managed timely manner
it just feels like don't care what you have to say when they do that
every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
Fun Monday. Reporting software we use broke. This software predates anyone that works in IT here and has been running without too much trouble for the last couple of years that I've worked here. Have had to restart the service every once in a while, or the data refresh doesn't run every so often and you have to run it manually, but otherwise, it's been ok. Uhtil the weekend (which no one noticed until Monday)
An Active Directory account that was from a guy that hadn't worked here in years that was long disabled broke it. I deleted that AD account as part of a cleanup on January 24th. On March 29th, 64 days later, the reporting refresh failed. It took two of us 6 hours to figure this out, as we had to learn how the software worked, and we eventually found this account had admin/full access to two data warehouse cubes that get refreshed twice a day, and because the account no longer existed in AD it just blew up and broke the whole thing.
No idea why this account would have had direct access, the admin access was service accounts otherwise.
Software licenses that are tied to hardware ID info can get fucked.
That's a pain with the Citrix stuff I deal with, but at least we can change it and redeploy them.
0
Options
lwt1973King of ThievesSyndicationRegistered Userregular
When I was moving off the on-premise Exchange server several years back, the transfer of the mailboxes was going fine. It was all good until I hit the one person who had over 10,000 emails in his inbox. Guess which one had corrupted emails so it would fail over and over again? Good times, good times.
Also, anyone have any experience with Juniper firewalls and their ease of use at just one location with VPN's pointing to about three different areas?
"He's sulking in his tent like Achilles! It's the Iliad?...from Homer?! READ A BOOK!!" -Handy
When I was moving off the on-premise Exchange server several years back, the transfer of the mailboxes was going fine. It was all good until I hit the one person who had over 10,000 emails in his inbox. Guess which one had corrupted emails so it would fail over and over again? Good times, good times.
Also, anyone have any experience with Juniper firewalls and their ease of use at just one location with VPN's pointing to about three different areas?
I do. Worked fine. But that was a very long time ago.
The only gotcha is if the other side of each VPN is Cisco. And that's more of a Cisco problem, because their whole lingo for configuring VPN tunnels is completely different from the rest of the industry.
every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
I'm so sick and tired of all these "secure email platforms" everyone's been using. I treat all of them as phishing attempts and 85% of the time I'm right on the money so why the fuck are people still using them?
I'm so sick and tired of all these "secure email platforms" everyone's been using. I treat all of them as phishing attempts and 85% of the time I'm right on the money so why the fuck are people still using them?
It's kind of sad (though I know why) that using public-private keys for emails on public email platforms isn't more of a norm these days. I remember setting that shit up in Thunderbird back in the day, but I'm pretty sure if I asked anyone younger about emailing me with my public key they would just get a confused look.
The thing with security is that we have to make it as easy as possible for users. It doesn't matter how secure something is if a user will not use it/bypass it. As much as a public/private key for email sounds easy when I look at 99% of my users here and if I asked them to use encrypted email that way it'd never fly. The fact that 2FA has even caught on and received pretty widespread adoption is a minor miracle, frankly.
This is why transparent things that users don't notice 99% of the time win. Things like conditional access, geo location gating, zero trust access, etc.
I'm so sick and tired of all these "secure email platforms" everyone's been using. I treat all of them as phishing attempts and 85% of the time I'm right on the money so why the fuck are people still using them?
The TLDR is that people expect to be able to get information by email and they don't know how email actually works and don't realize that information in emails isn't guaranteed to be secure during transit. Grandma would be pretty upset if she found out that her medical info was potentially sent in plain-text across the internet, but at the same time she wants to be able to get it by email and regular end users have no idea what encryption keys are.
I think my issue is that we keep setting users up to fail by coming up with systems that ask them to do the exact shit we tell them to stop fucking doing.
"Don't click on links in emails" is a pretty fucking big one that 99% of the time is right on the money. Yet banks, vendors, everyone will still send you emails with links or buttons in them that you need to click on, in order to view the actual contents of what they want to communicate. And it should not be that way.
They are training users to click on links in emails, and then pass credentials into a website, with absolutely no vision or understanding of what the fuck is going on or what this site is or if its real or wtf. It is the dumbest fucking bullshit, and it is some of the easiest trained behavior to exploit. Oh there's links and buttons that users are trained to click on and then pass credentials into? Surely nobody will come up with a way to harvest them.
I think my issue is that we keep setting users up to fail by coming up with systems that ask them to do the exact shit we tell them to stop fucking doing.
"Don't click on links in emails" is a pretty fucking big one that 99% of the time is right on the money. Yet banks, vendors, everyone will still send you emails with links or buttons in them that you need to click on, in order to view the actual contents of what they want to communicate. And it should not be that way.
They are training users to click on links in emails, and then pass credentials into a website, with absolutely no vision or understanding of what the fuck is going on or what this site is or if its real or wtf. It is the dumbest fucking bullshit, and it is some of the easiest trained behavior to exploit. Oh there's links and buttons that users are trained to click on and then pass credentials into? Surely nobody will come up with a way to harvest them.
This right here.
If you don't want your users clicking random shit then don't use systems that send them random shit.
The big challenge I see is that we use MFA, registration, and password reset links everywhere.
I'm so sick and tired of all these "secure email platforms" everyone's been using. I treat all of them as phishing attempts and 85% of the time I'm right on the money so why the fuck are people still using them?
It's kind of sad (though I know why) that using public-private keys for emails on public email platforms isn't more of a norm these days. I remember setting that shit up in Thunderbird back in the day, but I'm pretty sure if I asked anyone younger about emailing me with my public key they would just get a confused look.
Counterpoint:
I remember setting up a VC for encrypted emails with a bank, and they sent us both private and public keys in an open email. We replied gently that this would obviously not work, that we recommend they delete these keys and send us a new one, but only the public one this time.
The bank told us they couldn't delete the pair they sent, because that's the one they used for all their clients...
I think my issue is that we keep setting users up to fail by coming up with systems that ask them to do the exact shit we tell them to stop fucking doing.
"Don't click on links in emails" is a pretty fucking big one that 99% of the time is right on the money. Yet banks, vendors, everyone will still send you emails with links or buttons in them that you need to click on, in order to view the actual contents of what they want to communicate. And it should not be that way.
They are training users to click on links in emails, and then pass credentials into a website, with absolutely no vision or understanding of what the fuck is going on or what this site is or if its real or wtf. It is the dumbest fucking bullshit, and it is some of the easiest trained behavior to exploit. Oh there's links and buttons that users are trained to click on and then pass credentials into? Surely nobody will come up with a way to harvest them.
"Don't click links in email" is good advice but I suspect it's self-defeating because, well, people send links via email. Not just for secure email portals, but invoicing portals, saas logins, etc. I don't know about you, but email is the primary way I (and other people I work with) send end users notifications about new systems. Like, say, HR during onboarding: "Here's the link to the timecard entry page, and here's the benefits page", etc.
As for the secure email portals, like Mimecast, Zix, Cisco Ironport, etc it sucks that there are so many of them but they're still the least bad option (IMO). Public/private key exchange is so far beyond the capability of 99% of users we might as well be talking about neurosurgery.
Feral on
every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
Usually our training is "don't click suspicious links in emails, and if there's ever any doubt, pick up the phone and call the person who sent it to you." (And give them some guidance on identifying suspicious links.)
That leaves open a judgement call for the end users, and therefore a probability of failure, but as a harm reduction method it seems to, well, be less bad than the alternatives.
TBF there is still a constant subpopulation of users who simply can't wrap their heads around it, and will fall for the phishing email every goddamn time. So I'm not presenting this as a panacea or anything.
Edit: and not a hill I will die on. This is based on anecdotes more than data.
Feral on
every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
I think my issue is that we keep setting users up to fail by coming up with systems that ask them to do the exact shit we tell them to stop fucking doing.
"Don't click on links in emails" is a pretty fucking big one that 99% of the time is right on the money. Yet banks, vendors, everyone will still send you emails with links or buttons in them that you need to click on, in order to view the actual contents of what they want to communicate. And it should not be that way.
They are training users to click on links in emails, and then pass credentials into a website, with absolutely no vision or understanding of what the fuck is going on or what this site is or if its real or wtf. It is the dumbest fucking bullshit, and it is some of the easiest trained behavior to exploit. Oh there's links and buttons that users are trained to click on and then pass credentials into? Surely nobody will come up with a way to harvest them.
"Don't click links in email" is good advice but I suspect it's self-defeating because, well, people send links via email. Not just for secure email portals, but invoicing portals, saas logins, etc. I don't know about you, but email is the primary way I (and other people I work with) send end users notifications about new systems. Like, say, HR during onboarding: "Here's the link to the timecard entry page, and here's the benefits page", etc.
As for the secure email portals, like Mimecast, Zix, Cisco Ironport, etc it sucks that there are so many of them but they're still the least bad option (IMO). Public/private key exchange is so far beyond the capability of 99% of users we might as well be talking about neurosurgery.
For internal stuff, yes I email links to individual people but usually it's part of an existing conversation. I can't remember the last time I emailed a link to all our users for something, I don't think I ever have, frankly.
For external stuff, we have a ~30-year-old policy to never email instructions to our customers about their services. We will always call. If you get an email that says it's from us, it's not, because we don't send emails. The only email you'll get from us is the one with your invoice, and it'll just be attached to the email.
Frankly, I'm glad none of our users have any cloud or Microsoft credentials that can be harvested, I have neither the time nor the resources to stay on top of that shit.
I think my issue is that we keep setting users up to fail by coming up with systems that ask them to do the exact shit we tell them to stop fucking doing.
"Don't click on links in emails" is a pretty fucking big one that 99% of the time is right on the money. Yet banks, vendors, everyone will still send you emails with links or buttons in them that you need to click on, in order to view the actual contents of what they want to communicate. And it should not be that way.
They are training users to click on links in emails, and then pass credentials into a website, with absolutely no vision or understanding of what the fuck is going on or what this site is or if its real or wtf. It is the dumbest fucking bullshit, and it is some of the easiest trained behavior to exploit. Oh there's links and buttons that users are trained to click on and then pass credentials into? Surely nobody will come up with a way to harvest them.
"Don't click links in email" is good advice but I suspect it's self-defeating because, well, people send links via email. Not just for secure email portals, but invoicing portals, saas logins, etc. I don't know about you, but email is the primary way I (and other people I work with) send end users notifications about new systems. Like, say, HR during onboarding: "Here's the link to the timecard entry page, and here's the benefits page", etc.
As for the secure email portals, like Mimecast, Zix, Cisco Ironport, etc it sucks that there are so many of them but they're still the least bad option (IMO). Public/private key exchange is so far beyond the capability of 99% of users we might as well be talking about neurosurgery.
For internal stuff, yes I email links to individual people but usually it's part of an existing conversation. I can't remember the last time I emailed a link to all our users for something, I don't think I ever have, frankly.
For external stuff, we have a ~30-year-old policy to never email instructions to our customers about their services. We will always call. If you get an email that says it's from us, it's not, because we don't send emails. The only email you'll get from us is the one with your invoice, and it'll just be attached to the email.
Frankly, I'm glad none of our users have any cloud or Microsoft credentials that can be harvested, I have neither the time nor the resources to stay on top of that shit.
The following is just an anecdote. Not an argument.
When I moved to Portland I opened a new bank account with a local credit union. Per usual, I got an email with a link to the online banking website to the Gmail account I use for businesses & shopping.
Google flagged it as possible phishing because the credit union's SPF record was all fucked up.
I very politely contacted them by phone, explained that I was a new member with some IT experience and wanted to report a problem with their email. Then I sent the customer service rep details on how to unfuck their SPF with the understanding they would forward it along to IT.
Two weeks later, they fixed it.
Not really an argument except for two minor points I guess
First, less importantly, it's an example of how heavily businesses rely on email
Second, and much more importantly, if a business is gonna rely on email, they gotta pay attention to their SPF at minimum and preferably their whole demarc/dkim situation. There's no good excuse for having a fucked SPF in the 21st century, especially not a financial institution.
every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
Also, one of the biggest techniques we teach users is
"Hold your mouse over the link and make sure the address goes somewhere you expect"
If I send a link to our tech support portal, by God it better actually go to support.company.com and not through emailer69.marketingpartner.behaviortracker.privacyviolator.com or shittyemaillinkscrubber.whogivesafuck.com
every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
Also, one of the biggest techniques we teach users is
"Hold your mouse over the link and make sure the address goes somewhere you expect"
If I send a link to our tech support portal, by God it better actually go to support.company.com and not through emailer69.marketingpartner.behaviortracker.privacyviolator.com or shittyemaillinkscrubber.whogivesafuck.com
Which is useful advice until you implement email security controls that use URL rewriting to send any clicked links through phishing verification/malware sandboxing first before allowing access. Then every hover link is 200 characters of unreadable garbage to the end user. But such technical controls are far better than hoping all your users can do the right thing by manually checking hover links so the tradeoff is worth it.
What we like to emphasize in training is even if such advice doesn't necessarily apply at work, such behaviors are useful for personal email as well.
SiliconStew on
Just remember that half the people you meet are below average intelligence.
Also, one of the biggest techniques we teach users is
"Hold your mouse over the link and make sure the address goes somewhere you expect"
If I send a link to our tech support portal, by God it better actually go to support.company.com and not through emailer69.marketingpartner.behaviortracker.privacyviolator.com or shittyemaillinkscrubber.whogivesafuck.com
Which is useful advice until you implement email security controls that use URL rewriting to send any clicked links through phishing verification/malware sandboxing first before allowing access. Then every hover link is 200 characters of unreadable garbage to the end user. But such technical controls are far better than hoping all your users can do the right thing by manually checking hover links so the tradeoff is worth it.
What we like to emphasize in training is even if such advice doesn't necessarily apply at work, such behaviors are useful for personal email as well.
I've found across a couple of places that this particular class of technical controls failed more often than the training and introduced "fun" new types of issues. That was exactly the type of system I was complaining about with "shittyemaillinkscrubber".
every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
Lemme clarify a bit; what I've found (twice now) is that the email link scrubbers don't work appreciably better than plain old web filtering and spam filtering, and tend to catch the same types of low-hanging fruit. Their false negatives are largely (entirely?) the same false negatives as web & spam filters.
If the email link scrubbers were as invisible as web and spam filters, I'd support their use out of defense in depth. But they don't seem to add enough additional value to be worth it.
Edit: Mimecast and Zix, BTW
Feral on
every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
DoD has probably the largest public key implementation in the world and significant amounts of infrastructure and personnel dedicated to facilitating key generation and distribution and they still regularly run into day to day problems making it work. PKI for the general public is an absolute pipe dream
0
Options
Inquisitor772 x Penny Arcade Fight Club ChampionA fixed point in space and timeRegistered Userregular
Lemme clarify a bit; what I've found (twice now) is that the email link scrubbers don't work appreciably better than plain old web filtering and spam filtering, and tend to catch the same types of low-hanging fruit. Their false negatives are largely (entirely?) the same false negatives as web & spam filters.
If the email link scrubbers were as invisible as web and spam filters, I'd support their use out of defense in depth. But they don't seem to add enough additional value to be worth it.
It's stupid because it removes any visibility for the end-user, and therefore robs them of agency (and ultimately, responsibility). I run into this at work all the time - I literally can't decipher any links that are in my emails, so I don't bother. There are plenty of phishing emails that would look almost exactly like any of the dozens of random HR/IT gibberish emails I get in a given day, so one of the primary ways I will have to determine if something is even remotely valid is to look at where it's directing me.
Hell, right now I have a ticket in our service desk to get removed from some kind of global spam list that tells me when someone requests access to resources which I don't control. Over the weekend I got 50 of the damn things, and they all looked the same. If I was actually managing those resources, I would not have the mental bandwidth to review every single request appropriately and would likely just spam click "Approve" on everything so I could get rid of them in my inbox.
Posts
https://www.bleepingcomputer.com/news/microsoft/microsoft-confirms-windows-server-issue-behind-domain-controller-crashes/
ah good, as if I didn't have anything else to do today...
On domain controllers, because Active Directory handles redundancy so well, I do them spaced out over the two weeks after patch Tuesday.
One of them always gets patched right away. That's the canary in the coal mine.
the "no true scotch man" fallacy.
Even ignoring the human element, what if you need to get a piece of equipment back there for a building repair or something?
Edit: now I have mental images of a poor tech guy caught up in cat5 cable in the 1 foot gap behind the servers, found several years after his death and desiccated like an ancient dead rat.
That was written by a dude in 1994 as a temporary solution as Microsoft was porting the UI it was developing for Windows 95 into NT4 until the "good" UI was done. the 32GB FAT32 format limit was an arbitrary number hard coded into the UI because it needed to be a number, 32GB was larger than anything you'd fine in 1994, and would work until a permanent UI was written.
It's still in use in 2024.
Kind of funny as hell that the "I'll just do this thing until I ca.....oh god it's in production now" is also a thing that happened to Microsoft, and not just every Small/Medium Business in the world
https://arstechnica.com/gadgets/2024/03/windows-current-disk-formatting-ui-is-a-30-year-old-placeholder-from-windows-nt/
After a solid 3 days fighting with it, I try a different machine, and it works the first time.
So I go back to the original machine, after some digging, whoever set this printer up approximately 8 years ago typo'd two things in the network settings, the subnet mask and default gateway. So it could communicate with anything on the last octet range, but nothing else. That worked fine since the machine and the exchange server were on the same subnet for some reason, but it couldn't talk to any other subnet or the internet.
It's been "working fine" for 8 years so nobody really noticed. Kill me.
We set up a... what used to be a connection filter, not sure what it is now, in Exchange Online so it will accept messages from that NAT IP.
While O365 does have an anonymous SMTP setup for that sort of situation we already had this SMTP server in place, so it's only been used for stuff that is off our internal network.
That server ran some local distribution groups, and about 30 users that we didn't want to pay for an Exchange online license for (people like warehouse/delivery guys who needed email and phones). But since it was Exchange 2010 and it's been out of support for a couple of years, Microsoft is starting to block it from talking to Exchange Online, hence I was finally able to convince the powers that be that I needed to carve out time to get us off of it.
Because it is 2010, even though I do have everything off of it, the process of getting it off my network is turning into a nightmare, since you do still need on premesis tools to manage exchange online in 2024. There are powershell tools you can use now to do this instead of needing a full (free) Exchange server set up for recipient management, but those only work if your exchange enviornment was on 2013 or later. So... here is what I need to do to decomission Exchange 2010:
1: upgrade the Exchange 2010 server from 2010 SP3 to the latest update which is SP3 update rollup 32. (the patch level of the server was a build from 2013, I need to be on a build from at least late 2015 to perform steps 2 and 3)
2: Install an exchange 2016 instance. This will upgrade the on prem Exchange schema from 2010 to 2016
3: enable co-existance of Exchange 2010 and 2016. (this is why step 1 is necessary, need to be on a a build that supports co-existence)
4: run the hybrid exchange wizard on exchange 2016 so Exchange Online talks to that server instead of the 2010 server.
5. I can then go through the process of decomissionsing the 2010 server.
6 (Optional step). Install the powershell tools that do allow you to manage exchange online users without a full exchange server in the enviornment. This would upgrade the on prem schema to 2019. But I may just keep the 2016 server around for ease of use since we can use gui tools to manage what we need to manage still.
Why, you ask do I not just install Exchange 2019? Well you can't enable coexistence between Exchange 2010 and 2019. You have to first go to 2016.
Kill me.
BTW, wunderbar, all that sounds reasonable and not too big of a deal. Yeah, it sucks shit that they left a 2010 server running for 14 years but doing a double upgrade to 2016 then 2019 shouldn't be a hair-puller.
the "no true scotch man" fallacy.
In this case all it meant was:
two load balancers
two Horizon connection servers
three Horizon virtualization hosts
with hyperconverged infra so there's redundant storage
not an enormous investment
Just an example of how HA can refer to simple redundancy in small environments. (I'm not arguing that other interpretations are wrong, just expressing how the phrase can be used.)
the "no true scotch man" fallacy.
yeah it's just a pain in the ass, and something I've been begging for the time to do for a year and a half and then we had a cyber security audit done and shockingly a Server 2008 R2 server Running a build of Exchange 2010 from 2013 was found to be a pretty big vulnerability in our network and suddenly leadership was very interested in having it be fixed yesterday.
oof, i know that feeling
besides the stress and frustration of having to do it NOW rather than in a well-managed timely manner
it just feels like don't care what you have to say when they do that
the "no true scotch man" fallacy.
Simpler details: https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78baad9e27
Well fuck - check your versions people.
An Active Directory account that was from a guy that hadn't worked here in years that was long disabled broke it. I deleted that AD account as part of a cleanup on January 24th. On March 29th, 64 days later, the reporting refresh failed. It took two of us 6 hours to figure this out, as we had to learn how the software worked, and we eventually found this account had admin/full access to two data warehouse cubes that get refreshed twice a day, and because the account no longer existed in AD it just blew up and broke the whole thing.
No idea why this account would have had direct access, the admin access was service accounts otherwise.
That's a pain with the Citrix stuff I deal with, but at least we can change it and redeploy them.
Also, anyone have any experience with Juniper firewalls and their ease of use at just one location with VPN's pointing to about three different areas?
I do. Worked fine. But that was a very long time ago.
The only gotcha is if the other side of each VPN is Cisco. And that's more of a Cisco problem, because their whole lingo for configuring VPN tunnels is completely different from the rest of the industry.
the "no true scotch man" fallacy.
It's kind of sad (though I know why) that using public-private keys for emails on public email platforms isn't more of a norm these days. I remember setting that shit up in Thunderbird back in the day, but I'm pretty sure if I asked anyone younger about emailing me with my public key they would just get a confused look.
This is why transparent things that users don't notice 99% of the time win. Things like conditional access, geo location gating, zero trust access, etc.
The TLDR is that people expect to be able to get information by email and they don't know how email actually works and don't realize that information in emails isn't guaranteed to be secure during transit. Grandma would be pretty upset if she found out that her medical info was potentially sent in plain-text across the internet, but at the same time she wants to be able to get it by email and regular end users have no idea what encryption keys are.
"Don't click on links in emails" is a pretty fucking big one that 99% of the time is right on the money. Yet banks, vendors, everyone will still send you emails with links or buttons in them that you need to click on, in order to view the actual contents of what they want to communicate. And it should not be that way.
They are training users to click on links in emails, and then pass credentials into a website, with absolutely no vision or understanding of what the fuck is going on or what this site is or if its real or wtf. It is the dumbest fucking bullshit, and it is some of the easiest trained behavior to exploit. Oh there's links and buttons that users are trained to click on and then pass credentials into? Surely nobody will come up with a way to harvest them.
This right here.
If you don't want your users clicking random shit then don't use systems that send them random shit.
The big challenge I see is that we use MFA, registration, and password reset links everywhere.
Counterpoint:
I remember setting up a VC for encrypted emails with a bank, and they sent us both private and public keys in an open email. We replied gently that this would obviously not work, that we recommend they delete these keys and send us a new one, but only the public one this time.
The bank told us they couldn't delete the pair they sent, because that's the one they used for all their clients...
"Don't click links in email" is good advice but I suspect it's self-defeating because, well, people send links via email. Not just for secure email portals, but invoicing portals, saas logins, etc. I don't know about you, but email is the primary way I (and other people I work with) send end users notifications about new systems. Like, say, HR during onboarding: "Here's the link to the timecard entry page, and here's the benefits page", etc.
As for the secure email portals, like Mimecast, Zix, Cisco Ironport, etc it sucks that there are so many of them but they're still the least bad option (IMO). Public/private key exchange is so far beyond the capability of 99% of users we might as well be talking about neurosurgery.
the "no true scotch man" fallacy.
That leaves open a judgement call for the end users, and therefore a probability of failure, but as a harm reduction method it seems to, well, be less bad than the alternatives.
TBF there is still a constant subpopulation of users who simply can't wrap their heads around it, and will fall for the phishing email every goddamn time. So I'm not presenting this as a panacea or anything.
Edit: and not a hill I will die on. This is based on anecdotes more than data.
the "no true scotch man" fallacy.
Oh yeah. Fuck that. Looking at you, Cisco.
the "no true scotch man" fallacy.
For internal stuff, yes I email links to individual people but usually it's part of an existing conversation. I can't remember the last time I emailed a link to all our users for something, I don't think I ever have, frankly.
For external stuff, we have a ~30-year-old policy to never email instructions to our customers about their services. We will always call. If you get an email that says it's from us, it's not, because we don't send emails. The only email you'll get from us is the one with your invoice, and it'll just be attached to the email.
Frankly, I'm glad none of our users have any cloud or Microsoft credentials that can be harvested, I have neither the time nor the resources to stay on top of that shit.
The following is just an anecdote. Not an argument.
When I moved to Portland I opened a new bank account with a local credit union. Per usual, I got an email with a link to the online banking website to the Gmail account I use for businesses & shopping.
Google flagged it as possible phishing because the credit union's SPF record was all fucked up.
I very politely contacted them by phone, explained that I was a new member with some IT experience and wanted to report a problem with their email. Then I sent the customer service rep details on how to unfuck their SPF with the understanding they would forward it along to IT.
Two weeks later, they fixed it.
Not really an argument except for two minor points I guess
First, less importantly, it's an example of how heavily businesses rely on email
Second, and much more importantly, if a business is gonna rely on email, they gotta pay attention to their SPF at minimum and preferably their whole demarc/dkim situation. There's no good excuse for having a fucked SPF in the 21st century, especially not a financial institution.
the "no true scotch man" fallacy.
"Hold your mouse over the link and make sure the address goes somewhere you expect"
If I send a link to our tech support portal, by God it better actually go to support.company.com and not through emailer69.marketingpartner.behaviortracker.privacyviolator.com or shittyemaillinkscrubber.whogivesafuck.com
the "no true scotch man" fallacy.
Which is useful advice until you implement email security controls that use URL rewriting to send any clicked links through phishing verification/malware sandboxing first before allowing access. Then every hover link is 200 characters of unreadable garbage to the end user. But such technical controls are far better than hoping all your users can do the right thing by manually checking hover links so the tradeoff is worth it.
What we like to emphasize in training is even if such advice doesn't necessarily apply at work, such behaviors are useful for personal email as well.
I've found across a couple of places that this particular class of technical controls failed more often than the training and introduced "fun" new types of issues. That was exactly the type of system I was complaining about with "shittyemaillinkscrubber".
the "no true scotch man" fallacy.
If the email link scrubbers were as invisible as web and spam filters, I'd support their use out of defense in depth. But they don't seem to add enough additional value to be worth it.
Edit: Mimecast and Zix, BTW
the "no true scotch man" fallacy.
It's stupid because it removes any visibility for the end-user, and therefore robs them of agency (and ultimately, responsibility). I run into this at work all the time - I literally can't decipher any links that are in my emails, so I don't bother. There are plenty of phishing emails that would look almost exactly like any of the dozens of random HR/IT gibberish emails I get in a given day, so one of the primary ways I will have to determine if something is even remotely valid is to look at where it's directing me.
Hell, right now I have a ticket in our service desk to get removed from some kind of global spam list that tells me when someone requests access to resources which I don't control. Over the weekend I got 50 of the damn things, and they all looked the same. If I was actually managing those resources, I would not have the mental bandwidth to review every single request appropriately and would likely just spam click "Approve" on everything so I could get rid of them in my inbox.