Duplicate posts are an issue that seem to constantly plague autoblogging and similar autoposting WordPress plugins. There have been numerous causes for this problem in the past and we are constantly working on methods to fix this. We have had good success combating this problem, but occasionally some server configuration or WordPress update will reintroduce this bug.

Background

When AutoBlogged parses items in a feed, the first thing it does is grab the item’s title and link. Next, it will take those two values and check the database to see if those posts already exist in your WordPress blog. AutoBlogged will search the database for duplicate titles and/or links, depending on which options you have checked on the Filtering Options page.

When it searches for duplicate titles, it tries searching for the literal title as well as the sanitized form of the title using the WordPress sanitize_title and the  sanitize_title_with_dashes functions. If duplicate link checking is enabled, it will search the database for the exact link the post uses. Depending on which options you have enabled, it could do up to five checks for each post.

AutoBlogged should get one of three responses from the database: either the title and/or link already exists, an empty recordset that indicates they don’t exist, or an error message that something failed while processing the request.

The problem is that sometimes AutoBlogged gets back an empty recordset when a post exists rather than a positive result or an error message.

Causes

Most often the reason the database returns a false result is because some error or timeout occurred but the database did not return an error message. We have found this to be the case with some web server configurations. The problem is that by not returning an error, it is very difficult to debug the problem.

Another possible cause is that the character set that WordPress uses does not match that of the database therefore the duplicate checks never match any titles.

Solutions

To solve the problem with duplicates, we suggest you first make sure you have the latest AutoBlogged release. We are always improving the dupe checking code and this might be all you need to fix the problem. If that doesn’t do it, try the following solutions in order:

1. Try optimizing (or even repairing) all of your database tables using phpMyAdmin or your hosting control panel. About half the time this fixes the duplicates problem.
2. Check to make sure your database is not overloaded. If you ever see database errors on your WordPress site, this is very likely what is causing duplicate posts as well. You might also want to check the runtime configuration and system variables for possible problems using phpMyAdmin. There are links on the phpMyAdmin main page to show this information. There are a lot of settings here that affect MySQL performance and you may need an expert to help you out here if you suspect this is the problem.
3. Check the resource load on the server itself to make sure it isn’t overworked or maxing out its resources. This is fairly common with cheap shared hosting accounts.
4. If you have 10,000 or more posts in your blog or your site gets more than 3,000 visitors per day, you may simply need more powerful hardware. WordPress can be sluggish when there are too many posts or when it gets too busy and this could cause database connection failures.
5. Try configuring WordPress to use the default database character set by opening wp-config.php and removing or commenting out this line:

define(‘DB_CHARSET’, ‘utf8′);

If none of these solutions work for you, perhaps the easiest solution is to find a different hosting company. We actually very rarely see the problem on any of our test servers and we have noticed that it often occurs with cheap or oversold hosting companies. We have always had good results with HostNine and A Small Orange.

If you manage your own web server, you may need to find a MySQL expert to help you optimize the database for your type of load.

As we mentioned before, this is a problem that we are constantly monitoring and always working to eliminate. If you are unable to fix the problem yourself, we are happy to help you debug it, especially if it helps us find new fixes to prevent it.

…

Tags: autoblogged-problem duplicate-posts feed-to-blog feedwordpress-duplicates import-feeds WordPress Plugin wp-o-matic-duplicates

If you are seeing posts with extended characters such as †or  or � there are several things that can cause this. The most common causes are

• Multiple encoding of the same characters, most often seen with quotes and apostrophes.
• Mismatched input and output encoding of certain extended characters, most often seen with curly quotes, long dashes (em-dash), ellipses, etc.

Often you will see these appear when you are combining feeds from multiple sources through an external aggregater or external filter such as Yahoo pipes. Misconfigured web sites as well as some WordPress themes and plugins can also cause this. You might also see these characters appear with certain character sets shown on certain browser versions.

Character encoding problems are very common and are often seen in feed aggregators such as Google Reader.

The Problem

The characters appear because the current character set encoding incorrectly displays a character value. There are a number of different character sets used in data processing and each character set has unique availability and numbering of localized and international characters. ASCII, ISO 8859-2, Windows-1250, and UTF-8 are all common character sets, each with their own encoding schemes.

Because each character encoding is different a value encoded with one character set will display incorrectly with another character set. Since many character sets are extensions of ASCII, most of the basic Latin characters will show up correctly but will not correctly convert special characters such as curly quotes or long dashes. So while a long dash might appear correctly with one character set, in another character set it might be an upside down exclamation point, an accented letter, or not even exist at all.

Here is an example of incorrect character set conversion:

Blog » Blog Archive » Example Post

The more common problem is when a character is encoded or escaped for processing but never converted back. Because certain characters have special meaning with programming languages, SQL statements, scripting languages, and browsers, most often you will see text escaped or encoded before processing. When escaped or encoded text is escaped or encoded again, the result is an incorrect character when decoded.

Here is an example of double encoding or escaping:

‘This Title€™ Isn’t Displaying Correctly

Unfortunately, there is no easy solution that we can do on our end to fix this problem that will work for everyone. We can’t just filter out those characters because that messes up non-English characters that are meant to be there. We can’t do a simple charset conversion because we don’t always know the original charset and we also don’t know if the text has already been converted one or more times. Furthermore, some characters such as curly quotes simply don’t exist in common character sets such as ISO-8859-1.

To make matters worse, when you aggregate multiple feeds using something like Yahoo! Pipes, there could be multiple character sets combined in a single feed. And finally, the problem could be occurring after AutoBlogged processes the post by a mismatched character set in the database, web server, or WordPress theme.

Here are a few things you can try that might help:

• First, make sure the problem occurs on actual posts. Sometimes these characters show up when you preview the feed processing, but not on the blog itself.
• Try a different WordPress theme.
• Try the plugin mentioned here: http://www.prelovac.com/vladimir/ultimate-solution-to-weird-utf-cha… or here:http://kaloyan.info/blog/proekti/wordpress-proekti/wp-utf8-sanitize-plugin/
• Check your feeds at feedvalidator.org and also make sure the charset is correct.
• Try specifically defining your web site’s output character set with html meta tags.
• Open your wp-config.php file, find this line: define(‘DB_CHARSET’, ‘utf8′); and comment it out like this: //define(‘DB_CHARSET’, ‘utf8′);

Ultimately, you may find that some feeds just aren’t usable which does happen.

One solution we have resorted to in some extreme cases is to do a simple search and replace from within AutoBlogged. To do this, copy the unwanted character to the search for field and leave the replace with field empty.

We expect to have a working solution for this problem by our next release.

If you are a Twitter user and want to add posts to your blog for each of your tweets, AutoBlogged can help you out. Twitter provides an API that returns RSS feeds that you can enter into AutoBlogged.

There are several Twitter RSS feeds available, depending on what you want to show. These feeds are as follows:

Public Timeline

Description: Returns the 20 most recent status updates from non-protected users who have set a custom user icon.

http://­twitter.com/statuses/public_timeline.rss

Friends Timeline

Description: Returns the 20 most recent status updates posted by yourself and that your friends. Note that this feed requires authentication by replacing <username> and <password> with your own username and password in the URLs below.

http://<username>:<password>@twitter.com/statuses/friends_timeline.rss

Optional Parameters:

• since.  Optional.  Narrows the returned results to just those status updates created after the specified HTTP-formatted date, up to 24 hours old.  The same behavior is available by setting an If-Modified-Since header in your HTTP request.  Example: http://<username>:<password>twitter.com/statuses/friends_timeline.rss?since=Tue%2C+27+Mar+2007+22%3A55%3A48+GMT
• since_id.  Optional.  Returns only statuses with an ID greater than (that is, more recent than) the specified ID.  Example: http://<username>:<password>twitter.com/statuses/friends_timeline.rss?since_id=12345
• count.  Optional.  Specifies the number of statuses to retrieve. May not be greater than 200.  Example: http://<username>:<password>twitter.com/statuses/friends_timeline.rss?count=5
• page. Optional. Example: http://<username>:<password>twitter.com/statuses/friends_timeline.rss?page=3

User Timeline

Description:  Returns your 20 most recent status updates or the updates from the user specified by the ID. Note that this feed requires authentication by replacing <username> and <password> with your own username and password in the URL below.

The RSS feed for your own posts is as follows:

http://<username>:<password>@twitter.com/statuses/user_timeline.rss

To get the feed for another users’ post, use either their user number or user name like this:

http://<username>:<password>twitter.com/statuses/user_timeline/21336467.rss or http://twitter.com/statuses/user_timeline/autoblogged.rss

Note that the username and password are not required if you are pulling the timeline of another user that is not configured as protected.

Optional Parameters:

• count.  Optional.  Specifies the number of statuses to retrieve. May not be greater than 200.  Example: http://<username>:<password>twitter.com/statuses/user_timeline.rss?count=5
• since.  Optional.  Narrows the returned results to just those statuses created after the specified HTTP-formatted date, up to 24 hours old.  The same behavior is available by setting an If-Modified-Since header in your HTTP request.  Example: http://<username>:<password>twitter.com/statuses/user_timeline.rss?since=Tue%2C+27+Mar+2007+22%3A55%3A48+GMT
• since_id.  Optional.  Returns only statuses with an ID greater than (that is, more recent than) the specified ID.  Example: http://<username>:<password>twitter.com/statuses/user_timeline.rss?since_id=12345
• page. Optional. Example: http://<username>:<password>twitter.com/statuses/user_timeline.rss?page=3

Note that you can view the full Twitter API documentation here: http://apiwiki.twitter.com/REST-API-Documentation

Post Templates

Since a Twitter update is just a single line of text, the content in the feed is the same as the title. For that reason, you might want to leave the post template empty for Twitter feeds. However, if you do want to use a post template, here are some variables that might be useful:

%link% – This is the URL that points to the individual tweet on the web.

%source_url% – This is the URL that points to the archive you are pulling from.

%icon% – The twitter icon.

Here is an example post template that might work well for Twitter:

<p><img src=”%icon%” alt=”Twitter” /><a href=”%link%”>Twitter update</a>:  %description%</p>
<p>More here: <a href=”%source_url%”>%source_url%</a></p>

Note that beginning with AutoBlogged v2.4.23, you will be able to change the title of a post by using the Custom Fields feature. To do this, edit the feed, scroll down to the Custom Fields setting, and enter title in the Custom Field box then something like “Twitter Update” in the Custom Field Value box.

One thing we do here to be proactive with our technical support is we monitor search engine results for autoblogged-related errors. By watching to see what errors the search engines are indexing we can identify bugs that haven’t yet been reported. And since we are always looking for new uses for AutoBlogged, we monitor these errors with our own internal autoblog site.

To get comprehensive results, we use several tools. First, we need to get RSS feeds from various search engines, which we build using one of these tools:

http://alp-uckan.net/free/monitorthis/
http://www.researchbuzz.org/tools/kebberfegg.pl

What these tools do is let you enter in some search terms and it will generate the RSS feed URLs for various search engines in OPML format. Although AutoBlogged doesn’t as of yet allow importing OPML, we copy and paste each feed URL one at a time.

Another way to gather search results is to use a Yahoo! Pipes meta search such as this one here:

http://pipes.yahoo.com/pipes/pipe.info?_id=nHNB8TJm3BGumlGA9YS63A

Our internal support autoblog doesn’t just watch for errors, we also watch to see who is asking questions about our script, who is illegally sharing our script, who is writing about our script, and of course, who is praising it. Each group of feeds goes into its own category and it is all nicely organized, automatically tagged, and presented in a great-looking WordPress blog.

AutoBlogged is a powerful autoblogging plugin for WordPress that automatically creates blog posts from any RSS or Atom feed. Autoblogging is a great way to automate your WordPress posts so you can focus your efforts on earning money with your blog. With dozens of features, AutoBlogged is one of the most powerful autoblog software plugins available for WordPress and the best way to get automated blog content.

Features:

• Image and video support
• Custom post templates
• Advanced post filtering
• Enhanced tagging engine
• Regular Expression Search & Replace
• Create thumbnails for images
• Override feed data with your own values
• Fully supports WordPress 2.7 and later
• See AutoBlogged’s features

Furthermore, we have added a comprehensive online help, e-mail support, and customer forums to make sure you can quickly and easily get started autoblogging in WordPress! You can use AutoBlogged to build a blog network, as an automated video blog, to create topic portals, or aggregate RSS feeds. WP Autoblogs are a great way to quickly build keyword-dense content, earn money with your blog, and fully automate your blog posting. Even better, automated blog content is an excellent alternative to domain parking. Rather than showing a generic parked page with spammy-looking ads, you can provide real content, get indexed in search engines, build page rank, and generate traffic. You will be amazed how fast your domain value will grow!

Screenshots

More AutoBlogged screenshots…

RSS Feed Sources

• Configure multiple RSS feed sources with the ability to enable or disable individual feeds.
• Support for RSS 0.9, 0.91, 0.92, 1.0, 2.0, and Atom 0.3 and 1.0 feeds.
• Integrated feed caching, HTTP Conditional GET support, and support for GZIP-compression to improve performance and reduce bandwidth usage.
• Support for dozens of RSS modules including Dublin Core, GeoRSS, ITunes, Media RSS, RSS 1.0 Content, W3C WGS84 Basic GEO, XML 1.0, and XHTML 1.0.
• Numerous pre-defined searches to locate articles via Google Blog Search, Technorati, Blogdigger, Blogpulse, MSN Spaces, Yahoo! News, Flickr, YouTube, and others.
  
• RSS Feed autodiscovery–if you don’t know the exact feed URL, just enter the page address and AutoBlogged will find it for you.
• Built-in feed viewer to help with setting up and troubleshooting feed sources.
• Ability to override the automatically extracted feed data, such as author or source, with your own static values or values.
• Set the default post status to Published, Pending, Draft, or Private

Feed Processing

• Process feeds automatically using built-in pseudo cron feature so an external scheduler is not required.
• Set a range of intervals for random update scheduling.
• Manually process all feeds or one feed at a time.
• Preview feed processing without importing any posts.
• Disable cron operations without disabling the entire plugin.
• Allow other feeds to notify your autoblog of updates using an XML-RPC ping.
• Configure individual feeds to update every time AutoBlogged runs or after every each number of runs you set.
  
• Include all posts from each feed, set a limit on posts added per feed, or have AutoBlogged select random posts based on a percentage you set.
• Automatically create text-only excerpts based on number of words, sentences, or paragraphs.

Post Filtering

• Domain blacklist blocks posts from certain domains. Excellent way to block sites with spammy content, invalid HTML, or to allow webmasters to exclude their site from your blog.
• Block posts based on any portion of the URL to prevent posts from a specific type of sites such as forums or to block feeds from certain types of software.
• Keyword blacklists to exclude posts that contain keywords you specify.
• Duplicate post checking based on title and/or URL.
• Automatic filtering of malicious content in posts including SQL injection and cross-site scripting.
• Feed-specific filtering based on all words, any words, exact phrase, or none of the words specified.
  
• Feed-specific search and replace features using regular expressions to rewrite words, URLs, fix invalid content, replace affiliate IDs, etc.
• Truncate or filter out posts with long titles.
• Filter out posts where the titles are in all caps or where they contain multiple consecutive exclamation points or other punctuation.

Categories and Tags

• Assign each feed to one or more blog categories or subcategories or have AutoBlogged randomly select from a list of categories you specify.
• Search each post for existing blog categories and add them as additional categories or as tags on the post.
• Visits the original URL to extract additional tags not found in the feed using our own powerful tagging engine–an important SEO feature that will load your blog with related keywords.
  
• Add extra tags using the Yahoo! tagging API.
• Include categories from the original post and add missing categories to your blog if you choose.
• Tag blacklists prevent certain tags from appearing on a post.
• Common tags list increases the frequency of popular tags used by Technorati and other tagging sites or modify the list to add important tags for your niche.
• Provide a list of tags to randomly add to each post to increase the density of long tail phrases and other targeted keywords.
• Set the maximum and minimum tag length to ensure consistency and readability of your tags.
• Set the maximum number of tags to add to any post.

Authors

• Specify the author to use for new posts, assign a rando m author or use the name of the original author.
• If the original author does not exist as an author on your blog, add it, skip the post, randomly pick another author, or specify a default author.

• Use additional author information from your blog when adding new posts.

Post Templates

• Post templates let you randomly select from one or more post formats to ensure variety and to accommodate any number of site requirements.
• Apply different post templates to each of your feeds.
• Insert variables from the post, original feed, or any values you define.
• Random Select Lists to add variety to each post.
• Conditional Select Lists to show alternate fields if one is empty.
• Include images, video, flash and other content in your blog posts using an embedded video players.
• Specify a custom player for playing FLV or MP3 files.
• Build custom post templates to use with affiliate and other non-standard feed formats.
• Include text-only summaries or entire feed content as your post.
• Add custom HTML to each post to include NoFollow tags, Javascript, or even WordPress quick tags.
• Custom fields allow you to create your own post variables and include items as additional custom fields fin Wordpress.
• Automatic image, logo, and favicon extraction.
• Include any elements from the original feed based on dozens of supported RSS modules.
• And of course, you can simply show the original feed content untouched.

WordPress Integration

• Takes advantage of internal tagging and category engines to ensure strong keyword coverage and site navigation.
• Full integration with WordPress security and user permission features.
• Attribute posts to WordPress authors when incoming posts use that author’s name.
• Uses site information from WordPress blogroll when that site already exists in your blogroll.
• Automatically adds fields for images, videos, and thumbnails to support a variety of premium themes.

Other Features

• Set the HTTP Referer and User-Agent strings to use when visiting the original sites to advertise your site or provide opt-out instructions or contact info to other webmasters.
• Huge speed and performance improvements over previous versions.
• Commented PHP source code included.
• Support forum and e-mail support for all registered users.
• Free minor version updates included.

System Requirements

• PHP v4.3.2 or later
• WordPress v2.5 or later
• SimplePie Core Plugin v1.1 or later
• PHP cURL extension required
• PHP Zlib extension recommended

Note that servers with safe_mode enabled or an open_basedir set will have reduced image retrieval and tagging capabilities.

If you don’t want to read the entire documentation, below is a quick guide for setting up a simple topic portal blog. Before getting started, make sure you have the latest version of WordPress installed and working properly. You may find their Getting Started Guide helpful.

If you plan on using other plugins, you might first want to wait until you get AutoBlogged working properly to ensure there are no conflicts.

Configuring WordPress

There are a few basic things you need to do to get a minimal blog up and running. You will want to select a theme and fine-tune the settings for your blog. For this demonstration, however, we will just need to edit the default category.

From the admin area of your blog (usually located at http://yoursite.com/wp-admin), click on Manage, then on Categories. Now click on the default category and rename it to something that relates to your topic. Also create any extra categories you might want for you blog.

Installing AutoBlogged

To install the AutoBlogged plugin, extract the contents of the download file and upload the entire autoblogged directory to the /wp-content/plugins directory on your web host.

Note that AutoBlogged will attempt to set permissions on the /cache directory to 666. If that setting does not work, you may have to use chmod 777, which may pose a security risk. If you have trouble with this step, please contact your web hosting provider.

From the WordPress admin area, click on the Plugins menu, find AutoBlogged on the list and click on the link to enable the plugin.

Configuring AutoBlogged

1. Click on the Settings menu and then click on the AutoBlogged submenu
2. Click on the Add new feed link.
3. From the Feed Type list, select Google Blog Search.
4. In the Search Keywords box, enter the search terms you want for your subject.
5. Scroll down to the Categories and Tags section and select a default category for all posts from this feed.
6. Scroll down and click on Save Changes.
7. On the upper-left portion of the page, click on the Run Script Now link to test your settings and perform the first script run.

If the script runs correctly, you are finished. The script will automatically check for new posts every 1-3 hours.

What To Do Next

• Add more feeds
• Identify common search terms and long-tail phrases using tools such as Google’s Keyword Suggestion Tool. Add these in the Additional Tags option on the Tag Options screen.
• Review existing posts to see the tags collected and if you see any you don’t want, delete them and add them to the Tag Filtering box on the Tag Options screen.

AutoBlogged is a WordPress plugin that automatically builds content in your WordPress blog using one or more external RSS or Atom feeds. AutoBlogged is a fast yet powerful tool for building a wide variety of web sites, such as portals, news aggregators, topic watchers, popular article finder, autoblog gallery, or simply an alternative for domain parking.

Getting Started with AutoBlogged

Configuration and Usage

• Main Page
• Edit Feeds Page
• Tag Options Page
• Settings Page
• Filtering Page

Advanced Topics

• AutoBlogged Tagging Engine
• Working with Tags and Categories
• Post Template Reference

To send us your blog configuration, go to the AutoBlogged admin page in WordPress and select the Support subpage. From there you can send in a support ticket. Below the message field are two options you can check to include your configuration: Attach system version info and Attach entire AutoBlogged configuration.

Note that if you already have an active ticket open with our support, you can reference that ticket number in the subject (i.e., ticket #2200).

Issue

By default WordPress will send a notification to any external blog posts referenced in your site, which show up as comments to the original post.  Sometimes you may not want to notify other blogs when you autoblog their posts.

Solution

To disable pingbacks and trackbacks, go to the Discussion settings in WordPress and uncheck the option Attempt to notify any blogs linked to from the article (slows down posting.)