Status of Twitter (api/analysis/visualization) in Pharo?
Hi, I want to do some interaction with the Twitter API (retrieve tweets based on hashtags mostly) and visualize results of some analyses using Roassal. I saw there are some projects on Smalltalkhub, but none has much/any/up-to-date documentation. What is the current status of these projects? Does anyone know about any up-to-date documentation or examples? I am thankful for all hints. Best regards, Martin. -- View this message in context: http://forum.world.st/Status-of-Twitter-api-analysis-visualization-in-Pharo-... Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
Hi, We're working on a project that involves retrieve tweets and visualize results using Roassal also. Our approach includes data scraping and is based on the idea that public political discourse in covered under constitutional law which is above Twitters API Terms and Conditions. Is related with data activism and slow data and doesn't use Twitter's API. Scraping is a sensible issue but I think that we're covered by fair use and a lawyer working on digital rights thinks the same. Scraping happens at client level using chrome/chromium, without any use of Pharo for that. Pharo is used for extracting structured information of the tweets and visualizing it. Most of the detailed raw documentation is in Spanish as a product of our local workshops, but you can get and idea of what the project is doing at [1] to see is this approach works for you. [1] http://mutabit.com/offray/static/blog/output/posts/visualizing-politicianspo... Next week we well be doing a week long data visualization workshop [2], so I think that we will have better visualization, scraping algorithms and documentation (in Spanish). I will keep you posted on the general of how our workshop advances and sharing preliminary results. [2] http://mutabit.com/dataweek/ Cheers, Offray On 15/06/15 09:12, MartinW wrote:
Hi,
I want to do some interaction with the Twitter API (retrieve tweets based on hashtags mostly) and visualize results of some analyses using Roassal.
I saw there are some projects on Smalltalkhub, but none has much/any/up-to-date documentation. What is the current status of these projects? Does anyone know about any up-to-date documentation or examples?
I am thankful for all hints.
Best regards, Martin.
-- View this message in context: http://forum.world.st/Status-of-Twitter-api-analysis-visualization-in-Pharo-... Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
Thank you very much, Offray, that looks very interesting. When was the last time, you used the scraping plugin? It does run and scrolls down the twitter pages, but there is no result at all to be found in the console. Can it be that Twitter changed the format lately? I will try to find a way to harvest tweets first, and then I will try your visualization engine. Martin. -- View this message in context: http://forum.world.st/Status-of-Twitter-api-analysis-visualization-in-Pharo-... Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
Hi Martin, On 16/06/15 10:14, MartinW wrote:
Thank you very much, Offray, that looks very interesting.
Thanks. I think that is important to keep politicians and public accountable for what they say and do and social networks could help in doing that in a more dynamic way (of course is not enough).
When was the last time, you used the scraping plugin? It does run and scrolls down the twitter pages, but there is no result at all to be found in the console. Can it be that Twitter changed the format lately?
The last time was April, but if the scroll works, you should be able to inspect the DOM from the browser and export it following the instructions at the blog post. The browser knows about the DOM of the page that is being showed currently and if you are capable of seeing it in the browser, you should be able to inspect and save. We will test the script again and improve the code next week in a week long workshop on data visualization using Pharo/Roassal, so I will keep you posted about what's happening with this approach.
I will try to find a way to harvest tweets first, and then I will try your visualization engine.
Ok. Let me know what happens. Cheers, Offray
Offray wrote
Ok. Let me know what happens.
Hi Offray, I got the scraping, importing and visualization working. Not sure, what went wrong the first time. I found one little bug, when the avatar was a jpeg. It can be solved by changing the following lines in TwitterProfile>>scrapAvatarFrom: from: ((avatarUrl asLowercase endsWith: '.jpeg') or: (avatarUrl asLowercase endsWith: '.jpg')) ifTrue: [ avatarImage := ZnEasy get: avatarUrl. "(PNGReadWriter on: avatarImage ) nextPutImage: (JPEGReadWriter on: avatarImage )" ]. to: ((avatarUrl asLowercase endsWith: '.jpeg') or: (avatarUrl asLowercase endsWith: '.jpg')) ifTrue: [avatarImage := ZnEasy getJpeg: avatarUrl.]. Very nice result. Thank you a lot for sharing it! Best regards, Martin. -- View this message in context: http://forum.world.st/Status-of-Twitter-api-analysis-visualization-in-Pharo-... Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
Hi Martin, Thanks, we had already detected the problem with Jpeg images and even reported here to the list, but until now the solution eluded us. Your solution is already on the repo. Cheers, Offray On 21/06/15 10:53, MartinW wrote:
Offray wrote
Ok. Let me know what happens. Hi Offray, I got the scraping, importing and visualization working. Not sure, what went wrong the first time.
I found one little bug, when the avatar was a jpeg. It can be solved by changing the following lines in TwitterProfile>>scrapAvatarFrom:
from:
((avatarUrl asLowercase endsWith: '.jpeg') or: (avatarUrl asLowercase endsWith: '.jpg')) ifTrue: [ avatarImage := ZnEasy get: avatarUrl. "(PNGReadWriter on: avatarImage ) nextPutImage: (JPEGReadWriter on: avatarImage )" ].
to:
((avatarUrl asLowercase endsWith: '.jpeg') or: (avatarUrl asLowercase endsWith: '.jpg')) ifTrue: [avatarImage := ZnEasy getJpeg: avatarUrl.].
Very nice result. Thank you a lot for sharing it! Best regards, Martin.
-- View this message in context: http://forum.world.st/Status-of-Twitter-api-analysis-visualization-in-Pharo-... Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
Offray wrote
Thanks, we had already detected the problem with Jpeg images and even reported here to the list, but until now the solution eluded us. Your solution is already on the repo.
Cool :) -- View this message in context: http://forum.world.st/Status-of-Twitter-api-analysis-visualization-in-Pharo-... Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
I succeeded in installing parts of TwitterRoassalPlayGround in a Pharo 3 image. http://smalltalkhub.com/#!/~arturozambrano/TwitterRoassalPlayGround - The RESTRetriever seems to work fine, but the Rest results are very limited due to Twitter's own limitations. - The StreamingRetriever does simply not produce any results at all, but also no errors. -- View this message in context: http://forum.world.st/Status-of-Twitter-api-analysis-visualization-in-Pharo-... Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
MartinW wrote
I succeeded in installing parts of TwitterRoassalPlayGround in a Pharo 3 image. http://smalltalkhub.com/#!/~arturozambrano/TwitterRoassalPlayGround
- The RESTRetriever seems to work fine, but the Rest results are very limited due to Twitter's own limitations. - The StreamingRetriever does simply not produce any results at all, but also no errors.
I got the StreamingRetriever to work.. -- View this message in context: http://forum.world.st/Status-of-Twitter-api-analysis-visualization-in-Pharo-... Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
Hi Martin, this are good news. There is a seaside base UI for controlling tweets download. I will let you know as soon as it is uploaded. Visualizations are working, the code is crappy but works... I have some documentation I will send it to you (all of them are very basic examples we built for the last world cup), unfortunately it is Spanish only, but you will get the idea from the code snippets. regards art On Tue, Jun 16, 2015 at 12:44 PM, MartinW <wm@fastmail.fm> wrote:
MartinW wrote
I succeeded in installing parts of TwitterRoassalPlayGround in a Pharo 3 image. http://smalltalkhub.com/#!/~arturozambrano/TwitterRoassalPlayGround
- The RESTRetriever seems to work fine, but the Rest results are very limited due to Twitter's own limitations. - The StreamingRetriever does simply not produce any results at all, but also no errors.
I got the StreamingRetriever to work..
-- View this message in context: http://forum.world.st/Status-of-Twitter-api-analysis-visualization-in-Pharo-... Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
Thanks Arturo. Seems a really interesting advance. The seaside UI uses the Twitter API? Let us know when the project is uploaded. Cheers, Offray On 16/06/15 11:18, Arturo Zambrano wrote:
Hi Martin, this are good news. There is a seaside base UI for controlling tweets download. I will let you know as soon as it is uploaded.
Visualizations are working, the code is crappy but works... I have some documentation I will send it to you (all of them are very basic examples we built for the last world cup), unfortunately it is Spanish only, but you will get the idea from the code snippets. regards art
On Tue, Jun 16, 2015 at 12:44 PM, MartinW <wm@fastmail.fm <mailto:wm@fastmail.fm>> wrote:
MartinW wrote > I succeeded in installing parts of TwitterRoassalPlayGround in a Pharo 3 > image. > http://smalltalkhub.com/#!/~arturozambrano/TwitterRoassalPlayGround <http://smalltalkhub.com/#%21/%7Earturozambrano/TwitterRoassalPlayGround> > > - The RESTRetriever seems to work fine, but the Rest results are very > limited due to Twitter's own limitations. > - The StreamingRetriever does simply not produce any results at all, but > also no errors.
I got the StreamingRetriever to work..
-- View this message in context: http://forum.world.st/Status-of-Twitter-api-analysis-visualization-in-Pharo-... Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
participants (3)
-
Arturo Zambrano -
MartinW -
Offray Vladimir Luna Cárdenas