These are plenty of people worried about this. Just one example https://openai.com/index/preparing-for-future-ai-capabilitie...

This type of research requires experimentation (mostly failures) on extremely complex real-world equipment. Same with the nuclear weapons. AI being able to magically figure it out without experimental grounding is pure and absolute fantasy, used by companies like OpenAI and Anthropic as a justification for monopolizing AI R&D. In a sense it's not surprising this idea comes from rationalism-adjacent folks, as rationalism is mostly about the idea that experimentation is irrelevant and you can infer anything using just logic alone.

> In a sense it's not surprising this idea comes from rationalism-adjacent folks, as rationalism is mostly about the idea that experimentation is irrelevant and you can infer anything using just logic alone.

Yeah IIRC Yudkowski famously said something about a super intelligence could derive the theory of gravity correctly by seeing only three frames of a video depicting an apple falling from a tree. This is the same Less Wrong nonsense, rejecting how vital and irreplaceable experimentation is.

There's an infinite number of explanations for the location of an object in three equally time-spaced instances. Not to mention limitations of the measuring equipment itself.

There has been a lot of research into discovering new physics (starting by discovering old physics) since the last 5-6 years and it always require:

- A lot of high-quality data - Some careful design - (Not always) some external knowledge to guide the solutions

And this is using specialized NNs for physics, where you often know underlying equations. Kind of crazy that some people are so delusional about that.

Okay, I’m inclined to agree there.. but I can’t find the reference now, I also read that the worry is that the complexity required is coming down fast, and while maybe it’s not going to happen in a basement , there could be a small scale lab, not subject to rigorous certifications and checks that offers crisper as a service, and ai could be used to .. just perturb a protein a little bit so it doesn’t trigger some known virus black list, and people could just be ordering things online.

>rationalism is mostly about the idea that experimentation is irrelevant and you can infer anything using just logic alone.

Thanks for putting it the way you did. I didn't knew it was meant be that way, but it sort of confirms my suspicion that people who use the term 'rational' and 'logic' loosely often to dismiss an opposing view never really seek experimental results before having a point of view.

Viruses are perhaps the 3rd or 4th thing down the list of scary biological things people might make in their basement to end the human world.

> If a a virus is so deadly, everything it touches dies soon, it would not spread quickly but die out. If it is very contagious .. but very, very slow incubation time, so it infects the whole world, before becoming a deadly disease ..

This is a made up equilibrium that actually does not need to exist in nature.

Viruses and bacteria can in fact be both extremely, extremely contagious and extremely, extremely lethal.

> If a a virus is so deadly, everything it touches dies soon,

Trivially: you actually can have a virus that kills everything it touches not soon. Nothing in biology or chemistry or physics prevents it.

> Viruses and bacteria can in fact be both extremely, extremely contagious and extremely, extremely lethal.

Sure, but those two things would tend to work against it becoming a pandemic— unless it managed those two things but also kept its host healthy enough for long enough before becoming lethal to adequately spread it.

I looked into this once, it depends on how splashy the death is. A virus that made people explode instantly into a fine mist of airborne virus particles could be perfectly adequate for a pandemic (although holding off until help arrives might work even better).

"A virus that made people explode instantly into a fine mist of airborne virus particles could be perfectly adequate for a pandemic"

And what existing virus comes close to this trait?

I think we can safely assume that OP was picking a bit of a ridiculous hypothetical example to make a point that it’s possible for something to be deadly and transmissible, although in nature Baculovirus in Caterpillars has a similar mechanism (encourages their host to eat a lot, then climb to the top of a plant so when it turns to ooze it infects others) or cordyceps although both of these aren’t as highly transmissible as they hypothetical explode virus.

But the Black Death mixed high contagion and high mortality as an actual example that shows they aren’t mutually exclusive.

Oh, I would never say biological weapons are harmless, but the wiping out humanity claim I debated.

What? That's your second strawman in two comments.

Nobody said you claimed they were harmless. People are taking issue with your assertion that biological agents can be either contagious or lethal (not both), and therefore you discount its risk. This implied tradeoff between contagiousness and lethality simply is not enforced by anything in nature.

The natural emergence of a pathogen that's both highly contagious and highly lethal would be a much rarer event than the natural emergence of one that's either contagious or lethal, but we're talking about engineered pathogens. There is no reason to think that pathogens cannot be deliberately created that are both of those things.

None of you have seen ‘The Beauty’, I’m guessing.

No, but I have learned that sometimes there is a difference between fiction and reality.

I do understand your sentiment. But also, this isn't a party

> unless it managed those two things but also kept its host healthy enough for long enough before becoming lethal to adequately spread it.

I am clearly referring to this specific scenario. There is nothing in chemistry or biology or physics that prevents it.