Our EOL stuff is mostly due to mergers and ERP archives where they never want to pay for the data conversion to the current system, do not want to continue to pay for software maintenance of the old system to keep it updated or have support for migrating it to a newer OS, but they still need to occasionally reference accounting data for the next decade. All we can really do is network isolate and restrict account access to the extent possible and even keep the systems powered off unless a request comes in to access them temporarily.
Otherwise it's the occasional vendor-supplied equipment who still think XP is all the OS you'll ever need and no their stuff won't run without full admin permissions and you'd be foolish to ask. Though we've mostly fixed this over the last couple years through IT's efforts to convince management that million dollar shipments and severe penalties for their delay dependent on a decade old PC isn't a great business practice.
SiliconStew on
Just remember that half the people you meet are below average intelligence.
I mostly agree but I also think MFA is close to universally required now. Turns out using two related text string hashes as your only security is iffy.
Coincidentally, we don't currently have MFA on our hosted chat system. It's not Slack or Teams, but it's very similar and it's provided by our cloud VOIP company. It works very well.
...usually. Today, users are intermittently getting MFA prompts when attempting to log in to it. We didn't enable MFA on it (yet) but it's been on our radar.
Yay! Cloud!
every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
Coincidentally, we don't currently have MFA on our hosted chat system. It's not Slack or Teams, but it's very similar and it's provided by our cloud VOIP company. It works very well.
...usually. Today, users are intermittently getting MFA prompts when attempting to log in to it. We didn't enable MFA on it (yet) but it's been on our radar.
Getting requests from sysadmins with "senior" in their titles to help them install SSL certs on systems I don't manage was causing me to die a little inside.
every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
the "no true scotch man" fallacy.
+4
Options
RandomHajileNot actually a SnatcherThe New KremlinRegistered Userregular
Getting requests from sysadmins with "senior" in their titles to help them install SSL certs on systems I don't manage was causing me to die a little inside.
the only caveat I'll add to that is in the past when you could get a cert for 5 years it's just not something you had to do very often. Especially if you get a wildcard cert for a domain. I worked a job where managing certs was in my job description and I never ended up having to do it at that job because the certs didn't expire until about 6 months after I left there. So like... I didn't have to deal with certificate renewals for 6+ years.
Now that you can only get SSL certs for a year it should be a thing we deal with on a regular basis, so, you know, remember how to do it.
Getting requests from sysadmins with "senior" in their titles to help them install SSL certs on systems I don't manage was causing me to die a little inside.
the only caveat I'll add to that is in the past when you could get a cert for 5 years it's just not something you had to do very often. Especially if you get a wildcard cert for a domain. I worked a job where managing certs was in my job description and I never ended up having to do it at that job because the certs didn't expire until about 6 months after I left there. So like... I didn't have to deal with certificate renewals for 6+ years.
Now that you can only get SSL certs for a year it should be a thing we deal with on a regular basis, so, you know, remember how to do it.
Kinda. Renewals, absolutely. But you don't need CSR's until you get new certs. So people still end up out of practice.
Someone was asking me the other day how to generate a CSR on Windows.
So I'm sure you heard some laughter off in the distance that day. Sorry. That was me.
Like most procedures, renewing certs is solved with a bit of documentation.
This.
Everything I need to do once per year I make a page for in our team OneNote so that I don't have to slog through finding the one blog that actually explained it correctly, only to find it's now a Parked Domain
Like most procedures, renewing certs is solved with a bit of documentation.
i would love it if i could get people to consistently read and follow documentation
let alone apply a modicum of independent thought when reality diverges from the documentation
"what does invalid key size mean?"
(it means you skipped the part in the documentation that told you to change the key size from the default 1024 to 2048)
"okay my cert still isn't working. halp."
(well, I just looked up the knowledgebase for the software that I am not the SME in, you're the supposed SME in, and it explicitly says that certificates must be in PFX format, and yours is in PEM format. so maybe fix that.)
"how do I fix that?"
(redownload it from the CA)
"how do I do that?"
oh ffs...
Feral on
every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
documentation: How to Make a Peanut Butter & Jelly Sandwich
1) Open a jar of peanut butter.
2) Obtain a slice of bread.
3) Using a butter knife, apply peanut butter smoothly to one side of that slice of bread.
4) Place that slice of bread on a plate for now.
5) Open a jar of jelly.
6) Obtain a NEW slice of bread.
7) Using a butter knife, apply jelly smoothly to one side of the new slice of bread.
8) Put both slices of bread together.
Result:
"feral, halp. the users are complaining that their fingers are sticky!"
Feral on
every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
Getting requests from sysadmins with "senior" in their titles to help them install SSL certs on systems I don't manage was causing me to die a little inside.
the only caveat I'll add to that is in the past when you could get a cert for 5 years it's just not something you had to do very often. Especially if you get a wildcard cert for a domain. I worked a job where managing certs was in my job description and I never ended up having to do it at that job because the certs didn't expire until about 6 months after I left there. So like... I didn't have to deal with certificate renewals for 6+ years.
Now that you can only get SSL certs for a year it should be a thing we deal with on a regular basis, so, you know, remember how to do it.
The best practice now is that it should be scripted wherever you can do so.
Most (all?) of the major CAs have APIs now and instructions on how you can script the renewals.
I haven't gotten around to scripting them in my environment yet, but it's on the agenda.
But even if we can script it across all of our Windows and Linux VMs, there's still going to be oddball (virtual and physical) appliances and other oneoffs that will need to be manual. And for those, I'm not writing up documentation on each and every snowflake system, so I need people to understand the concepts and make independent decisions.
every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
We merged with another smaller org, 70ish users. They have a couple 2008 servers and a 2010 exchange server. Yikes. Ok, let's get them onto our 2016 on prem Exchange and then down the road we all go to 365. I'll need to pre-migrate all the email and then repoint the MX and then do my final migration. We want to blast the new remote agent onto all those machines so we just take a day, do them all by hand and say hello to all the users.
Oh, someone else controls your domain reg? Ok, let's take back control of that. Ok, so who controls your apparently Cloudflare DNS? You don't know? We spend 2 weeks digging and never find it? National head office who obviously controls it has no record and repeatedly states they don't? Cool, cool. Ok, we can add you to our DNS, get all the records we know of because you're small and your setup is very simplistic and then flip the name servers day of, and they'll propogate out over a few hours. Wait, didn't you have a cert on your OWA? I know you did, because the migration tool requires that. Oh, you had an undocumented reverse proxy setup in Cloudflare with the cert on it.
I've been running into too many instances of software throwing "System.OutOfMemoryException" errors recently. I've come across two different programs that use SQL on the backend doing it, and now a video wall program. Seems to have all started happening since mid-late December and all of the software vendors have no answers.
Because if you're going to attempt to squeeze that big black monster into your slot you will need to be able to take at least 12 inches or else you're going to have a bad time...
This VMware virtual appliance runs its own internal kubernetes cluster. During the setup process, the setup scripts configure the kubernetes cluster with its own internal IP schema, and there's an nginx instance inside kubernetes that accepts traffic from outside kubernetes.
I'm not knowledgeable about kubernetes, but I can at least tell that when I SSH into the VM, run any kind of networking tests from the OS, the networking works... except when trying to connect to the nginx instance running inside kubernetes.
This appliance is supposed to bridge the gap between a cloud VDI pod and an on-premises VDI pod. We have no cloud VDI. We only need this appliance for one stupid purpose: to enable the Horizon licenses we've already bought and paid for.
Yaaayyy cloud
It's so reliable
every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
VMware, by email: "We've opened a Zoom session for you. Please join at..."
Me, 15 minutes later, sees email. Attempts to join.
Immediately receives another email: "Zoom session closed due to lack of customer response."
This fucking industry.
This whole fucking industry.
every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
the good news is that on full VMs with real grown-up OSes you should be able to yum or apt yourself a new pkexec or polkit and be happy
you can also remove pkexec or polkit in a lot of cases and be happy
the problem is going to be, again, fucking virtual appliances and other stuff running some form of linux (possibly embedded, maybe iot) where the admin has to rely on the vendor for patching
every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
That_GuyI don't wanna be that guyRegistered Userregular
Filling out HIPAA audit forms is the most mind number experience imaginable. It's dozens of pages of questions like that, clearly written by a lawyer instead of a tech.
Filling out HIPAA audit forms is the most mind number experience imaginable. It's dozens of pages of questions like that, clearly written by a lawyer instead of a tech.
I've done HIPAA, Sarbanes-Oxley, and financial regulatory exams & auditing.
HIPAA is pretty bad. Financial industry audits are the same flavor, but even worse.
every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
Filling out HIPAA audit forms is the most mind number experience imaginable. It's dozens of pages of questions like that, clearly written by a lawyer instead of a tech.
I've done HIPAA, Sarbanes-Oxley, and financial regulatory exams & auditing.
HIPAA is pretty bad. Financial industry audits are the same flavor, but even worse.
The "I'm not storing unencyprted credit cards" 200 question quiz is probably the worst one to do every year.
not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
Whereas I sit here without such garbage, kinda wishing I had such garbage so that I could use it to enforce some sensible policy.
oh, i definitely prefer working in regulated industries.
the bureaucracy and paperwork is awful, specially for PCI, but at least regulated companies take cybersecurity and reliability seriously
at least, more often and more seriously than unregulated companies
i'd rather have a predictable but shitty PCI or HIPAA or Dept of Treasury exam once or twice a year than have to convince a computer-illiterate CEO why the latest ransomware outbreak was an indirect result of him cutting IT headcount down to me and an intern
Feral on
every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
Posts
Otherwise it's the occasional vendor-supplied equipment who still think XP is all the OS you'll ever need and no their stuff won't run without full admin permissions and you'd be foolish to ask. Though we've mostly fixed this over the last couple years through IT's efforts to convince management that million dollar shipments and severe penalties for their delay dependent on a decade old PC isn't a great business practice.
...usually. Today, users are intermittently getting MFA prompts when attempting to log in to it. We didn't enable MFA on it (yet) but it's been on our radar.
Yay! Cloud!
the "no true scotch man" fallacy.
sounds cloudy with a change of mfaballs today
Why would I trust you to know what the fuck you're doing with the keys I give you if you can't even generate a CSR?
Coincidentally I've been training folks in my department how to do CSRs.
Because until I started doing so, I was the only person in my IT department who understood how to generate a CSR and install a cert.
the "no true scotch man" fallacy.
the "no true scotch man" fallacy.
This is a clickable link to my Steam Profile.
the only caveat I'll add to that is in the past when you could get a cert for 5 years it's just not something you had to do very often. Especially if you get a wildcard cert for a domain. I worked a job where managing certs was in my job description and I never ended up having to do it at that job because the certs didn't expire until about 6 months after I left there. So like... I didn't have to deal with certificate renewals for 6+ years.
Now that you can only get SSL certs for a year it should be a thing we deal with on a regular basis, so, you know, remember how to do it.
Kinda. Renewals, absolutely. But you don't need CSR's until you get new certs. So people still end up out of practice.
Someone was asking me the other day how to generate a CSR on Windows.
So I'm sure you heard some laughter off in the distance that day. Sorry. That was me.
This.
Everything I need to do once per year I make a page for in our team OneNote so that I don't have to slog through finding the one blog that actually explained it correctly, only to find it's now a Parked Domain
i would love it if i could get people to consistently read and follow documentation
let alone apply a modicum of independent thought when reality diverges from the documentation
"what does invalid key size mean?"
(it means you skipped the part in the documentation that told you to change the key size from the default 1024 to 2048)
"okay my cert still isn't working. halp."
(well, I just looked up the knowledgebase for the software that I am not the SME in, you're the supposed SME in, and it explicitly says that certificates must be in PFX format, and yours is in PEM format. so maybe fix that.)
"how do I fix that?"
(redownload it from the CA)
"how do I do that?"
oh ffs...
the "no true scotch man" fallacy.
1) Open a jar of peanut butter.
2) Obtain a slice of bread.
3) Using a butter knife, apply peanut butter smoothly to one side of that slice of bread.
4) Place that slice of bread on a plate for now.
5) Open a jar of jelly.
6) Obtain a NEW slice of bread.
7) Using a butter knife, apply jelly smoothly to one side of the new slice of bread.
8) Put both slices of bread together.
Result:
"feral, halp. the users are complaining that their fingers are sticky!"
the "no true scotch man" fallacy.
The best practice now is that it should be scripted wherever you can do so.
Most (all?) of the major CAs have APIs now and instructions on how you can script the renewals.
I haven't gotten around to scripting them in my environment yet, but it's on the agenda.
But even if we can script it across all of our Windows and Linux VMs, there's still going to be oddball (virtual and physical) appliances and other oneoffs that will need to be manual. And for those, I'm not writing up documentation on each and every snowflake system, so I need people to understand the concepts and make independent decisions.
the "no true scotch man" fallacy.
We merged with another smaller org, 70ish users. They have a couple 2008 servers and a 2010 exchange server. Yikes. Ok, let's get them onto our 2016 on prem Exchange and then down the road we all go to 365. I'll need to pre-migrate all the email and then repoint the MX and then do my final migration. We want to blast the new remote agent onto all those machines so we just take a day, do them all by hand and say hello to all the users.
Oh, someone else controls your domain reg? Ok, let's take back control of that. Ok, so who controls your apparently Cloudflare DNS? You don't know? We spend 2 weeks digging and never find it? National head office who obviously controls it has no record and repeatedly states they don't? Cool, cool. Ok, we can add you to our DNS, get all the records we know of because you're small and your setup is very simplistic and then flip the name servers day of, and they'll propogate out over a few hours. Wait, didn't you have a cert on your OWA? I know you did, because the migration tool requires that. Oh, you had an undocumented reverse proxy setup in Cloudflare with the cert on it.
...
Are you kidding me?
the "no true scotch man" fallacy.
Edit: It's BitDefender causing it: https://community.bitdefender.com/en/discussion/89709/out-of-memory-error
> curl -I https://www.vmware.com
HTTP/1.1 200 OK
> curl -I https://cloud.vmware.com
HTTP/1.1 200 OK
> curl -I https://fml.vmware.com
HTTP/1.1 200 OK
> curl -I https://cloud.fml.wtf.lol.bbq.vmware.com
HTTP/1.1 200 OK
> curl -I https://the.vmware.appliance.my.trouble.ticket.is.for.feral.com
Failed to connect to 10.69.4.20 port 443 after 1024 ms: Connection refused
VMware support: "The problem is that the appliance can't connect to the vmware cloud. please make sure your firewall is not blocking it."
the "no true scotch man" fallacy.
I'm not knowledgeable about kubernetes, but I can at least tell that when I SSH into the VM, run any kind of networking tests from the OS, the networking works... except when trying to connect to the nginx instance running inside kubernetes.
This appliance is supposed to bridge the gap between a cloud VDI pod and an on-premises VDI pod. We have no cloud VDI. We only need this appliance for one stupid purpose: to enable the Horizon licenses we've already bought and paid for.
Yaaayyy cloud
It's so reliable
the "no true scotch man" fallacy.
Me, 15 minutes later, sees email. Attempts to join.
Immediately receives another email: "Zoom session closed due to lack of customer response."
This fucking industry.
This whole fucking industry.
the "no true scotch man" fallacy.
Finally get on the Zoom call, the technician opens up a console session on the appliance and from the appliance's command line:
VMware technician: "This shows us that the appliance does not have external connectivity"
the "no true scotch man" fallacy.
Shit like this makes me able to get rid of my imposter syndrome.
https://nakedsecurity.sophos.com/2022/01/26/pwnkit-security-bug-gets-you-root-on-most-linux-distros-what-to-do/
i am so, so tired
the "no true scotch man" fallacy.
yeah but if you combine this with an arbitrary code execution technique, maybe one that is also very commonplace on linux like OH I DUNNO LOG4J
i'll be in a literal blanket and pillow fort with my cat and a teddy bear and a bottle of whiskey if anybody needs me
the "no true scotch man" fallacy.
I was complaining to my friend that computers were a mistake yesterday. I want to go hide in the woods.
you can also remove pkexec or polkit in a lot of cases and be happy
the problem is going to be, again, fucking virtual appliances and other stuff running some form of linux (possibly embedded, maybe iot) where the admin has to rely on the vendor for patching
the "no true scotch man" fallacy.
does our vulnerability scanner pick up pwnkit?
yes, dad
have you done a vuln scan for pwnkit?
yeessss, dad
are you mitigating...
yessss daaaaaaad
are your tracking the mitigations...
yyyyyyessssssss daaaaadddddd gawd leave me alone
the "no true scotch man" fallacy.
I've done HIPAA, Sarbanes-Oxley, and financial regulatory exams & auditing.
HIPAA is pretty bad. Financial industry audits are the same flavor, but even worse.
the "no true scotch man" fallacy.
The "I'm not storing unencyprted credit cards" 200 question quiz is probably the worst one to do every year.
oh, i definitely prefer working in regulated industries.
the bureaucracy and paperwork is awful, specially for PCI, but at least regulated companies take cybersecurity and reliability seriously
at least, more often and more seriously than unregulated companies
i'd rather have a predictable but shitty PCI or HIPAA or Dept of Treasury exam once or twice a year than have to convince a computer-illiterate CEO why the latest ransomware outbreak was an indirect result of him cutting IT headcount down to me and an intern
the "no true scotch man" fallacy.