Noob question: I have some 1D conv net for financial time series prediction. Could a transformer architecture be better for this task, is it worth a try?
If you think a longer context length might be helpful consider stacking convolutions to give higher units a bigger receptive field, or try the convolutional LSTM. If that helps and you have a further argument for why an even larger context window would be helpful then perhaps try attention and in that case a Transformer would be reasonable. But your stacked conv net would be the fastest and most obvious thing that should work (with the caveat that I know nothing else about your data and it's characteristics, which is a really big caveat).
Consider looking at your errors and judging whether they stem from things your current model doesn't do well but that Transformers do, i.e., correlating two steps in a sequence across a large number of time steps. Attention is basically a memory module, so if you don't need that it's just a waste of compute resources.
Thanks for the insight, also for mentioning convolutional LSTM, I wasn't aware such a thing existed.
> Attention is basically a memory module, so if you don't need that it's just a waste of compute resources.
But aren't CNNs also like a memory module (ie: they memorize how leopard skin looks like)? I guess attention is a more sophisticated kind of memory, "more dynamic" so to speak.
Anyway, I'm glad to hear that a transformer architecture isn't totally stupid for my task, I will look up the literature, there seems to be a bit on this matter.
Yeah, in some sense any layer is a "memory module". Perhaps more specifically, attention solves the problem of directly correlating two items in a sequence that are very, very far away from each other. I'd generally caution against using attention prematurely as it's extremely slow, meaning you'll waste a lot of your time and resources without knowing if it'll help. Stacking conv layers or using recurrence is an easy middle step that, if it helps, can guide you on whether attention could provide even more gains.
What scientists mean by "teleportation" is more like this: you have one particle A here, one particle B there, and by "teleportation" you make particle B have the same state as particle A.
Critical point: for each particle you want to "teleport", you have recipient particle standing by to receive it.
The particle ends up either getting destroyed during the measurement (e.g. that free photon or electron will no longer be), or its quantum state will have been collapsed into something that can no longer be used in a quantum algorithm.
Or how many Russians don't understand why Ukraine for example is upset.
Is as if propaganda works. Who would have thought.
Once I was discussing with a Chinese colleague and the Tiananmen subject came up (it was in the news, I didn't bring it up), and he told me that the students were not that innocent either, that they killed 150 soldiers the night before. I asked him, how could some students with rocks kill 150 soldiers with machine guns, but he said it was true. As suspected, I couldn't find any reputable sources for this later when I did extensive googling.
There is a lot of nuance there. Some soldiers were killed by protesters, but it wasn’t the student protesters rather beijingers who were pissed about the PLA coming into town (the PKU students were already mostly back in their dorms by this point). The soldiers also weren’t armed, this being early and back then they were afraid to just give the soldiers weapons (not all of the PLA agreed with the crackdown). They eventually brought in troops from further away garrisons fully armed that were less sympathetic to the protesters (local garrisons back then were staffed with soldiers from Beijing).
So your colleague wasn’t completely lying, but the truth belies a lot more intrigue. I think this is all in the wiki article.
The biggest problem (at least from the CPC’s point of view) is that the PRC completely lacked non lethal riot suppression capabilities. They literally had nothing in between just standing there unarmed and full on military assault. Afterwards, they built up the PAP (people’s armed police) very quickly, and you can see them often practicing in Beijing.
Maybe that's YOUR concept, but not everyone's. If you go to Wikipedia, you can look at citations from a variety of sources, which doesn't depend on Google. You could go to the website of various news organizations and search through their archives, which also does not depend on Google.
Sean's comment above about 'a lot of nuance' is a huge understatement.
First off, I think China is a corrupt superpower and a huge threat to the USA, and there is no comparison or ambiguity about who the good guys are (the USA).
But the Tiananmen Square thing was severely fucked up. Originally the soldiers were not allowed to fire on civilians, and they were lynched and burned alive. There are lots of photos of the corpses of the soldiers being displayed or placed in conspicuous places by the civilians.
The civilians could kill the soldiers because the soldiers were under orders not to hurt civs.
As a Chinese American, I've noticed that propaganda is everywhere and we all need to be wary of it.
Tiananmen is often brought up, but did you know that the US experienced a parallel event? I didn't learn about this in US history class, but apparently soldiers fired upon unarmed student protesters in the Kent State shooting: https://en.wikipedia.org/wiki/Kent_State_shootings
America has had so many shootings you can't really cover them all. For example the one I was never taught was the Greenwood Massacre despite its scale: https://en.wikipedia.org/wiki/Tulsa_race_riot
I wouldn't say it's taught in almost all high school history classes. In my experience the history curriculum doesn't cover very many things past world war 2. For example the New York State social studies curriculum has everything from 1945-1990 in a single unit, with the Vietnam war being a sub-unit ( http://www.nysed.gov/common/nysed/files/programs/curriculum-... )
Sorry, I forgot that America is the only country in the world where a certain number of people need to die by shooting for people to care, and that number has never been attained...
You're putting words in my mouth and totally misrepresenting my point.
I never said we shouldn't care about Kent State, nor have I said that it was unimportant.
What I said is that Tiananmen was several orders of magnitude worse, and while anyone in the US can look up Kent State and other horrible occurrences in their history books, many of them found in schools, or in any other place they'd like, people in China cannot look up the Tiananmen Square massacre.
I am replying to someone claiming that the US has done the same with Kent State as China has done with Tiananmen Square and that the events were equal to begin with. So if you can show me where the US has tried to cover up the fact that Kent State happened at even 1/1000th of the effort that China has gone through to cover up Tiananmen Square, or that the loss of human life between the two is similar, this might be a productive conversation.
In the mean time, you're responding to a point I never made and words I never said.
Wow I haven't seen those before. The reddit images were really powerful and disturbing. There's no arguing that the Chinese government has committed atrocities and is trying to cover them up.
My point is just that, I feel like the Western dialogue is often "Look at these horrible things that happen in China. Isn't propaganda sad?" My point is that propaganda is happening everywhere. Yes, it's happening in China blatantly and fragrantly. But everyone needs to be wary and alert that it might be happening even in seemingly "safe" countries.
For instance, I enjoyed your vimeo link and it compellingly documented the obvious censorship of the issue. The video made clear that some people clearly know what happened on 6/4 but it's not socially okay to talk about.
Interestingly, there is a small and consistent mistranslation in the subtitles. What the video translates as "don't record it", is "don't record me". Aka, some of the subjects take issue with the videographer recording people without permission. In addition to the sensitive topic, (I imagine) some part of their discomfort on screen is due to being randomly interviewed and recorded. But this was seemingly mistranslated in the videographer's favor? I point this out, not to argue with the message of the video, but to show that there is nuance/spin happening everywhere.
Thank you for taking the time to share these very interesting links. I'm grateful that I live in a place where it's possible to have this open and reasonable dialogue.
> My point is that propaganda is happening everywhere. Yes, it's happening in China blatantly and fragrantly. But everyone needs to be wary and alert that it might be happening even in seemingly "safe" countries.
Yes, I totally agree. To scapegoat the CCP as "the" evil power center would not only be unfair and hypocritical, it would be dangerous.
For me as a German, I a lot of beef with "my" country, "the West" etc. But that doesn't make me like Putin or the CCP, etc. If anything, the more corruption or tyranny there already is, the worse any additional "amount" of it becomes, if you know what I mean.
Of course that also means I cannot excuse "our" crimes with that of the CCP. For me that'd be like using the misdeeds of others to excuse one's own, which is actually worse than just doing the misdeed. And criticizing elsewhere is easy.
But so is looking the other way, and I also must not do that, I can't. If I want to secure and deserve what freedom I have, I have to support that of others, at least in principle, always.
> But this was seemingly mistranslated in the videographer's favor? I point this out, not to argue with the message of the video, but to show that there is nuance/spin happening everywhere.
No disagreement here either. And it really disheartens me when causes with legitimate grievances "give it a little extra". I wish people were more strict about that, it's a real problem. And making such small corrections, no matter how benign or factual, gets people labeled as "being in the other camp" so quickly, which is the worst part, apart from people thinking in terms of "sides" alll the time in the first place. So thanks for pointing this out.
You are free to talk about Kent state, it is covered in a lot of documentaries. There is even a monument at Kent State about it (as opposed to the “nothing happened here 6/4/89” plaque at Tiananmen Square). It isn’t taught in HS history class for the sake reason that Vietnam war is not taught (there is simply too much history and not enough time to cover it all).
A reasonable comparison might be Black Wall Street, but even then discussion about the event isn’t suppressed, just at the time the event was covered up by the local government of the area. But the CCP doesn’t remember the Siege of Changchun either.
Kent State is incredibly famous and covered in the context of the US's wars in Southeast Asia in any US History class. I definitely learned about it in high school. It also isn't actively scrubbed from the internet or other sources of information like Tiananmen square is in China. If you are unaware of Kent State, it isn't due to propaganda or censorship, it is due to your own ignorance.
It is the cover up - not the crime essentially. China shuts down stock markets when they come up with shared dates to Tiananmen Square. The US teaches it in textbooks and acknowledges impacts like radicalizing domestic terrorism groups. If not for premature detonation of an apartment it would have lead to another school massacre of a ROTC in retaliation.
We can't support 7 billion people if each one of them has a house with garden and grows it's own food. That's what the middle ages were, look at the population then.
The truth is probably somewhere between the two of you. We shouldn't be entirely dependant on centralized systems, nor should we attempt to be 100% self-sufficient. I'm sure there's a best of both worlds approach.
NYC was without power for 2 days in the Northeast Blackout of 2003 (the event the article was describing), and it generally brought people closer together. Folks went outside and chatted with their neighbors rather than hole up inside with nonexistent A/C to watch blank TV screens. A number of blocks organized impromptu barbecues to use up all the meat that was about to spoil, since grilling was the only cooking method available to many people.
It's a crapshoot which way society goes when the infrastructure disintegrates, but Lord of the Flies is definitely not a given. Historically people seem just as likely to come together as beat each other apart.
Two days is not in the same league as several weeks.
Plus New York City also had a blackout in 1977 that was very different.
>Looting and vandalism were widespread, hitting 31 neighborhoods. Possibly the hardest hit were Crown Heights, where 75 stores on a five-block stretch were looted, and Bushwick, where arson was rampant with some 25 fires still burning the next morning. At one point, two blocks of Broadway, which separates Bushwick from Bedford-Stuyvesant in Brooklyn, were on fire. Thirty-five blocks of Broadway were destroyed: 134 stores looted, 45 of them set ablaze. Thieves stole 50 new Pontiacs from a Bronx car dealership.[1] In Brooklyn, youths were seen backing up cars to targeted stores, tying ropes around the stores' grates, and using their cars to pull the grates away before looting the store.[1] While 550 police officers were injured in the mayhem, 4,500 looters were arrested.[1]... In all, 1,616 stores were damaged in looting and rioting. A total of 1,037 fires were responded to, including 14 multiple-alarm fires. In the largest mass arrest in city history, 3,776 people were arrested. Many had to be stuffed into overcrowded cells, precinct basements and other makeshift holding pens. A congressional study estimated that the cost of damages amounted to a little over $300 million (equivalent to $1.2 billion in 2017).
Weeks is a long time. As long as food and water can be minimally supplied, people can get along pretty well even in awful situations. Stuff only deteriorates once people can't feed themselves.
NYC is already reliant on others for food and water. Supplying it without electricity would be a severe logistics challenge but easier in some ways than in Puerto Rico.
