Forum Search:
Forum.Brain-Cluster.com: Brain Cluster Technical Forum
Ultimate forum for Technical Discussions

Home » Microsoft » Windows Server » Active Directory » Large Data Extract
Large Data Extract [message #161549] Wed, 30 September 2009 11:32 Go to next message
drm  is currently offline drm  United States
Messages: 14
Registered: September 2009
Junior Member
I have created a customer ADAM directory that consolidates customer
records from several applications. I am now asked to extract this
information, almost 3,000,000 records, from the directory so that it
can be loaded into a CRM application. Most attempts at dumping this
data into a CSV file have taken over 3 hours to run. Is there a
configuration, method or trick I can use that will allow me to extract
this data quicker? Are there security concerns with it?

Note: I will be moving the directory to newer hardware 1 qtr 2010, so
rebuilding it a certain may be an option.
Re: Large Data Extract [message #161551 is a reply to message #161549] Wed, 30 September 2009 12:46 Go to previous messageGo to next message
florian  is currently offline florian  Germany
Messages: 484
Registered: July 2009
Senior Member
Howdie!

drm schrieb:
> I have created a customer ADAM directory that consolidates customer
> records from several applications. I am now asked to extract this
> information, almost 3,000,000 records, from the directory so that it
> can be loaded into a CRM application. Most attempts at dumping this
> data into a CSV file have taken over 3 hours to run. Is there a
> configuration, method or trick I can use that will allow me to extract
> this data quicker? Are there security concerns with it?
>
> Note: I will be moving the directory to newer hardware 1 qtr 2010, so
> rebuilding it a certain may be an option.

Apart from moving the whole data to new hardware, there are a few things
you can do:
- try to separate the data into batches so they can be moved one at a time
- tailor the data down to what you really need to extract. Don't dump
anything (into an intermediate format) that won't be needed.
- check what export methods you have. Are there export formats that work
faster than others?

Cheers,
Florian
--
Microsoft MVP - Group Policy
eMail: prename [at] frickelsoft [dot] net.
blog: http://www.frickelsoft.net/blog.
Maillist (german): http://frickelsoft.net/cms/index.php?page=mailingliste
Re: Large Data Extract [message #161582 is a reply to message #161551] Thu, 01 October 2009 13:35 Go to previous message
drm  is currently offline drm  United States
Messages: 14
Registered: September 2009
Junior Member
On Sep 30, 2:46 pm, "Florian Frommherz [MVP]"
<flor...@frickelsoft.DELETETHIS.net> wrote:
> Howdie!
>
> drm schrieb:
>
> > I have created a customer ADAM directory that consolidates customer
> > records from several applications.  I am now asked to extract this
> > information, almost 3,000,000 records, from the directory so that it
> > can be loaded into a CRM application.  Most attempts at dumping this
> > data into a CSV file have taken over 3 hours to run.  Is there a
> > configuration, method or trick I can use that will allow me to extract
> > this data quicker?  Are there security concerns with it?
>
> > Note: I will be moving the directory to newer hardware 1 qtr 2010, so
> > rebuilding it a certain may be an option.
>
> Apart from moving the whole data to new hardware, there are a few things
> you can do:
> - try to separate the data into batches so they can be moved one at a time
> - tailor the data down to what you really need to extract. Don't dump
> anything (into an intermediate format) that won't be needed.
> - check what export methods you have. Are there export formats that work
> faster than others?
>
> Cheers,
> Florian
> --
> Microsoft MVP - Group Policy
> eMail: prename [at] frickelsoft [dot] net.
> blog:http://www.frickelsoft.net/blog.
> Maillist (german):http://frickelsoft.net/cms/index.php?page=mailingliste

Thanks.
Previous Topic:join system to domain
Next Topic:Trust between 2 2k3 domains in separate forests
Goto Forum:
  


Current Time: Wed Jan 17 05:52:05 MST 2018

Total time taken to generate the page: 0.02020 seconds
.:: Contact :: Home ::Sitemap::.

Powered by: FUDforum 3.0.0RC2.
Copyright ©2001-2009 FUDforum Bulletin Board Software