Do you have any details on the experiment procedures? E.g. hardware, training time, loss curves? It is difficult to confidently reproduce research without at least some of these details.
We provide the source code and weights along with a Docker environment to facilitate reproducing the experimental results.
The original paper’s EXPERIMENTS section mentions the hardware configuration (8× RTX 2080 Ti).
TFP cites decision transformers. Just using a transformer does not bypass the credit assignment problem. Transformers are an architecture for solving sequence modeling problems, e.g. the credit assignment problem as arises in RL. There have been many other such architectures.
The hardness of the credit assignment problem is a statement about data sparsity. Architecture choices do not "bypass" it.
The DT citation [10] is used on a single line, in a paragraph listing prior work, as an "and more". Another paper that uses DTs [53] is also cited in a similar way. The authors do not test or discuss DTs.
> hardness of the credit assignment ... data sparsity.
That is true, but not the point I'm making. "Bypassing credit assignment", in the context of long-horizon task modeling, is a statement about using attention to allocate long-horizon reward without horizon-reducing discount, not architecture choice.
To expand: if I have an environment with a key that unlocks a door thousands of steps later, Q-Learning may not propagate the reward signal from opening the door to the moment of picking up the key, because of the discount of future reward terms over a long horizon. A decision transformer, however, can attend to the moment of picking up the key while opening the door, which bypasses the problem of establishing this long-horizon causal connection.
(Of course, attention cannot assign reward if the moment the key was picked up is beyond the extent of the context window.)
You can do Q-Learning with a transformer. You simply define the state space as the observation sequence. This is in fact natural to do in partially observed settings. So your distinction does not make sense.
DT's reward-to-go vs. QL's Bellman incl. discount, not choice of architecture for policy. You could also do DTs with RNNs (though own problems w/ memory).
Saw a panel with the golem people just last night, and sure enough this question came up. The short answer is that they don't have a solution yet and IMO their thinking was no more advanced than what I'm seeing on this thread.
How is their problem substantially different from this project? Apart of course from the overheard and complexity caused by trying to force what should probably just be a centralised service into a blockchain.
Don't get me wrong, I think it's a great idea. I just don't see why it needs a blockchain and all the associated trustless infrastructure. Even nicehash doesn't bother with all that.
I suppose the approach by the Golem team is substantially different because the ideology associated with it.
What you see as “trying to force what should probably just be a centralized service,” I see as “innovating a new approach to powering decentralized architecture.”
I’m not saying you’re wrong. It would be easier to solve the problem using existing tool sets and more mature protocols. Yet, I’m pretty sure that the Golem team is doing something right. So there’s that. Maybe this isn’t a zerosum thing.
> I’m pretty sure that the Golem team is doing something right
I'm not at all convinced that the golem team have any particular insight to solve this obvious and common problem that everyone else doesn't have. And frankly I think that the overhead of running unnecessary infrastructure will render them price-uncompetitive to any reasonable centralised provider. In short, I predict they will fail.
But eh, they raised USD$8m and I didn't, so what do I know.
The Golem team doesn't necessarily need to solve every problem. Being built on top of the Ethereum Network is advantageous. If they make an appealing, open platform with potential, maybe other developers will pick up the ball and run with it to power their own ends.
> I guess that's why we're sitting in different camps.
Indeed, doesn't mean I don't want to hear the other side's point of view though!
Open source is not going to save them. They have one main problem - how to tell if people did the work they claim they did? If centralised, they can "test" new users or perhaps periodically check up on long term users by secretly allocating duplicate work and verifying its content. How can you do that in public? The blockchain is actually working against them.
And who really needs a cryptographically secure attestation that on march 25th 2018, user XYZ completed ML shared 456.7? This is a level of audit logging appropriate for a bank and basically nothing else. All you need is availability accounting of some sort. It's not rocket science. I couldn't write the client for this app but I sure as hell could write the back end and I wouldn't even think of using a blockchain. Make no mistake, their choice of technologies is for buzzword compliance, not technical necessity - a very bad sign.
There is also no need for the GNT. It solves no problem and users could just as easily be compensated in ETH or anything else. Sure, it's a funding mechanism, fine. We still haven't figured out how ICOs should even work.
Despite all the rigmarole, they have a product they need to sell like any other startup: rent us your GPU/CPU for $x/hr. Because of their overhead, I predict they will easily be outcompeted by centralised providers. People are not going to use golem over another, better-paying alternative just from the goodness of their hearts. And I cannot see any way how golem can be structurally more efficient than a centralised solution.
All said, I'm not as optimistic as you. Not like I want them to fail though, good luck to them!
Well I sure do appreciate you going out of your way to explain your perspective to me.
Just a couple more responses:
1. The use of blockchain is not for buzzword compliance. Julian (CEO) is a longtime Ethereum supporter/developer. This project has really been in development since 2014 or so, long before blockchain was "buzzy." So the use of blockchain here is not for grabbing cash. They actually think it's a better (perhaps harder) way forward.
2. Not only does the token allow investors to directly invest in the project, it also allows developers to "print" tokens that can be locked behind smart contracts. That way developers can be rewarded for reaching project goals with bowls of their own dog food. Not bad to eat when it's pretty much "real" money.
3. The decentralized and distributed nature of the project will allow the Golem Network to achieve goals and execute code that no centralized competitor could achieve/run. I'll leave it as a thought exercise for you to speculate what those goals/codes might look like.
Thanks for the engagement. It's great to test my beliefs through debate. Time will be the true arbitrator here though. Best wishes.
This model is showing predictions one day into the future. The "test set" plot is all predictions made with data from 1 day ago. The input sequences have size 1, so no recurrence is happening (see to_1dimension in https://github.com/miguelgfierro/sciblog_support/blob/master...). If you predict that tomorrow will have the same price as today you'll get better plots under the same operating conditions.
In[2]: TIME_AHEAD = 1
Train set has ~1e-6 MSE, Test set has ~0.8 (0.94^2).
EDIT: I should say this person is probably learning, and a lot of this is honest mistake.
Most of the research in compressed sensing type applications now is focused on more sophisticated prior distributions than sparse/Laplacian. These more general bayesian approaches provide significant performance improvements over LASSO etc.
reply