The Designer's Guide Community
Forum
Welcome, Guest. Please Login or Register. Please follow the Forum guidelines.
Mar 18th, 2024, 8:00pm
Pages: 1 2 
Send Topic Print
Virtuoso Slow Down after Large Memory exhaustion (Read 418 times)
cheap_salary
Senior Member
****
Offline



Posts: 162

Virtuoso Slow Down after Large Memory exhaustion
Dec 16th, 2017, 5:46am
 
I use very complex and very heavy post processing with ViVa in Cadence Virtuoso.
However I always encounter "Swap Activity Warning".

Once I encounter "Swap Activity Warning", Virtuoso is very slow down and never recover normal operation speed, even if I close ViVa.

I have to restart Virtuoso to recover normal operation speed.

I think it is because Virtuoso can not release allocated memory.

Is ther any method to recover normal operation speed without restarting Virtuoso ?
Back to top
 

2017-1215-1406-52.png
View Profile   IP Logged
Andrew Beckett
Senior Fellow
******
Offline

Life, don't talk to
me about Life...

Posts: 1742
Bracknell, UK
Re: Virtuoso Slow Down after Large Memory exhaustion
Reply #1 - Dec 21st, 2017, 7:34am
 
There has been some work on this recently - but in general releasing memory back to the operating system is challenging in most applications.

Please contact Cadence Customer Support for this as then a Cadence AE can see if there's anything that can be done to help (not sure which subversion of the IC tools you're using, for example).

Regards,

Andrew.
Back to top
 
 
View Profile WWW   IP Logged
cheap_salary
Senior Member
****
Offline



Posts: 162

Re: Virtuoso Slow Down after Large Memory exhaustion
Reply #2 - Dec 24th, 2017, 7:33am
 
My Subversion is:
Back to top
 

2017-1218-1652-13.png
View Profile   IP Logged
Andrew Beckett
Senior Fellow
******
Offline

Life, don't talk to
me about Life...

Posts: 1742
Bracknell, UK
Re: Virtuoso Slow Down after Large Memory exhaustion
Reply #3 - Dec 26th, 2017, 2:39am
 
OK - you're pretty recent (there's only one later subversion currently). I think you should talk to customer support, as I mentioned earlier - that way Cadence can get a better idea of what you're doing to see what is going on and to see if it can be improved.

Thanks,

Andrew.
Back to top
 
 
View Profile WWW   IP Logged
cheap_salary
Senior Member
****
Offline



Posts: 162

Re: Virtuoso Slow Down after Large Memory exhaustion
Reply #4 - Feb 6th, 2018, 4:32am
 
When I do large post processing by using Cadence Skill in IC5, I don't encounter an exhaustion of memory.
And I can complete in post processing.

However a large memory exhaustion occurs and it is never released, when I use Cadence Skill in IC6.
And I can not complete in post processing.

Here I don't use graph drawing, that is "ocean -nograph".

My conclusion is that used memories blow up if I use Cadence Skill in IC6.

I think this must be a critical defect of Cadence Skill in IC6.
Back to top
 
« Last Edit: Feb 6th, 2018, 9:39pm by cheap_salary »  
View Profile   IP Logged
Andrew Beckett
Senior Fellow
******
Offline

Life, don't talk to
me about Life...

Posts: 1742
Bracknell, UK
Re: Virtuoso Slow Down after Large Memory exhaustion
Reply #5 - Feb 8th, 2018, 12:59pm
 
That's a rather simplistic conclusion. It's not very likely to be SKILL itself that is the problem, but the objects being created not being garbage collected frequently enough.

Various types of objects are organised so that garbage collection is triggered when you run out of those objects, so it tries to do some housekeeping before allocating any more. It could be that your code is hanging onto objects which are no longer needed and thus preventing them being recycled, or it could be if this is waveform data that it is being cached to improve performance. You still read waveform data if you're doing simulation post-processing and not plotting anything; the caching occurs to improve performance if you access the same waveform more than once - we don't want to keep re-reading the same data from disk.

This is precisely why we (Cadence) need to see what you're doing - it's likely to be dependent on the nature of what you're doing rather than being something as simple as this being a "critical defect of Cadence Skill in IC6". Otherwise it would affect everyone, all the time. Which it doesn't.

Andrew.
Back to top
 
 
View Profile WWW   IP Logged
cheap_salary
Senior Member
****
Offline



Posts: 162

Re: Virtuoso Slow Down after Large Memory exhaustion
Reply #6 - Feb 9th, 2018, 3:07am
 
Andrew Beckett wrote on Feb 8th, 2018, 12:59pm:
That's a rather simplistic conclusion.
I don't think so, since I use same skill code and same psf data for both IC5 and IC6.

Andrew Beckett wrote on Feb 8th, 2018, 12:59pm:
It could be that your code is hanging onto objects which are no longer needed
and thus preventing them being recycled,
or it could be if this is waveform data that it is being cached to improve performance.
I use only four work variables, "Vrec_out_Raw", "Ain0_dBV", "yyy" and "y".
So these work spaces have to be reused.

Andrew Beckett wrote on Feb 8th, 2018, 12:59pm:
You still read waveform data if you're doing simulation post-processing
I do simulation as a completely another session, there I generate only psf data.

My skill script is for pure post processing without simulation.

Andrew Beckett wrote on Feb 8th, 2018, 12:59pm:
it's likely to be dependent on the nature of what you're doing
rather than being something as simple as this being a "critical defect of Cadence Skill in IC6".
I don't think so.

Andrew Beckett wrote on Feb 8th, 2018, 12:59pm:
Otherwise it would affect everyone, all the time. Which it doesn't.
Simply no one do heavy post processing by skill code.

Code:
Vrec_out_x = 1.5

for(i, 1, 150
  sprintf(psf_dir, "results_dir_%d/pss", i)
  openResults(psf_dir)

  selectResult("pss_fd")
  Vrec_out_Raw = real( harmonic( v("vout"), 0 ) )
  Ain0_dBV = cross(Vrec_out_Raw, Vrec_out_x, 1, 'rising)
  fprintf(fp, "%g", Ain0_dBV)

  selectResult("pss_td")
  for(k, 1, 200
    sprintf(yyy, "aho_%d:in", k)
    y = value( i(yyy), "Ain_dBV" Ain0_dBV )
    fprintf( fp, ",%g", average( abs(y) )/1m )

    sprintf(yyy, "boke_%d.Vb", k)
    y = value( v(yyy), "Ain_dBV" Ain0_dBV )
    fprintf( fp, ",%g", ymax(y) )
  ); for k

  when( dbGetDatabaseType() == "OpenAccess", closeResults(psf_dir) )
  fprintf(fp, "\n")
  drain(fp)
); for i

close(fp)

exit() 


I can complete this code by Skill in IC5.
But I can not by Skill in IC6.

If loop is small, e.g. " for(k, 1, 10" instead of " for(k, 1, 200" , I can complete even by Skill in IC6.

BTW, "closeResults(psf_dir)" is available in OCEAN in IC6,
but it is not helpful at all.
Back to top
 
« Last Edit: Feb 9th, 2018, 9:54pm by cheap_salary »  
View Profile   IP Logged
Andrew Beckett
Senior Fellow
******
Offline

Life, don't talk to
me about Life...

Posts: 1742
Bracknell, UK
Re: Virtuoso Slow Down after Large Memory exhaustion
Reply #7 - Feb 18th, 2018, 1:34am
 
Quote:
Simply no one do heavy post processing by skill code.


Of course, nobody has written any SKILL code in the last 12 years (since IC6.1.0 was released) which does any heavy processing. What a ridiculous statement.

I'm afraid you're incorrect about this being a critical defect in SKILL. What it's likely to be (as I said before) is something that is not freeing a cached waveform object. This is not necessarily related to the variables you're using in your script. It's likely to be related to the fact that you are reading 150+150*200*2 signals (60150) and there's also another temporary waveform created in the inner loop (150*200) for the abs function.

Yes, of course these should be garbage collected and the memory re-used, but there is a tradeoff to be had with respect to signals read in from disk - you don't want to have to keep reading the same data off disk over and over again, which happens surprisingly often. This is why you should report this to Cadence with the data and your script so that R&D can take a look and see if any of the recent improvements help - or if something else needs to be done to prevent memory being exhausted. Quite likely there is.

However, as I said before - this is not a critical defect in SKILL itself. You're conflating correlation and causality. It's due to a change in how waveform objects (in ViVA) are handled in Virtuoso between IC5 and IC6 - there are lots of other benefits of this change (performance in particular) but with all implementations there are tradeoffs, and clearly in this case there is the opportunity (if you report it) to improve that tradeoff in the specific situation you're facing.

Regards,

Andrew.
Back to top
 
 
View Profile WWW   IP Logged
cheap_salary
Senior Member
****
Offline



Posts: 162

Re: Virtuoso Slow Down after Large Memory exhaustion
Reply #8 - Feb 18th, 2018, 1:44am
 
Andrew Beckett wrote on Feb 18th, 2018, 1:34am:
What it's likely to be (as I said before) is something that is not freeing a cached waveform object.
Right.

Andrew Beckett wrote on Feb 18th, 2018, 1:34am:
It's likely to be related to the fact that you are reading 150+150*200*2 signals (60150)
and there's also another temporary waveform created in the inner loop (150*200) for the abs function.
Right.

However "closeResults(psf_dir)" should release memories.

"closeResults(psf_dir)" was introduced in IC6 which does not exist in IC5.
But it is no meaning at all.

Conclusion is that skill in IC6 consume very large memories than IC5 and never release them.
http://www.designers-guide.org/Forum/YaBB.pl?num=1513431975/0#0
I have to restart Virtuoso to recover normal operation speed.

So we have to make one session of skill very small in IC6.

However ability of treatment is too small in IC6.

Due to this limitation, ADE-XL, ADE-GXL and ADE-Explorer assume to use multiple sessions of skill.
Back to top
 
« Last Edit: Feb 18th, 2018, 3:05am by cheap_salary »  
View Profile   IP Logged
Andrew Beckett
Senior Fellow
******
Offline

Life, don't talk to
me about Life...

Posts: 1742
Bracknell, UK
Re: Virtuoso Slow Down after Large Memory exhaustion
Reply #9 - Feb 18th, 2018, 1:54pm
 
Again, you're drawing conclusions based on incorrect assumptions.

  • The purpose of closeSession is to free up the ADE session (and release the license); the goal is not necessarily to free memory (although this might be a side benefit of a recent improvement in the latest versions of IC617 - not sure if it helps your case though, hence the request to go to customer support). Might be worth checking with IC617 ISR17 first, although I can't guarantee that will help solve your problem.
  • ADE XL, Explorer and Assembler do not make any assumptions about using "multiple sessions of skill". They actually use several ICRPs (the background process used in ADE XL, Assembler and Explorer) in order to process in parallel (parameterisation, netlisting, simulation control and post-processing). It's not done for memory reasons.
  • It certainly does re-use memory for waveform objects in many cases given the kind of processing that many ADE users do, but clearly that's not happening properly in your specific case.


Have you actually contacted Cadence customer support over this issue?

Regards,

Andrew.
Back to top
 
 
View Profile WWW   IP Logged
cheap_salary
Senior Member
****
Offline



Posts: 162

Re: Virtuoso Slow Down after Large Memory exhaustion
Reply #10 - Feb 18th, 2018, 4:52pm
 
Andrew Beckett wrote on Feb 18th, 2018, 1:54pm:
The purpose of closeSession is to free up the ADE session
Wrong.

I use closeResults() which are introduced in IC6.

Back to top
 
 
View Profile   IP Logged
Andrew Beckett
Senior Fellow
******
Offline

Life, don't talk to
me about Life...

Posts: 1742
Bracknell, UK
Re: Virtuoso Slow Down after Large Memory exhaustion
Reply #11 - Feb 19th, 2018, 3:11am
 
Quote:
Wrong.

I use closeResults() which are introduced in IC6.

My mistake. I misread closeResults as ocnCloseSession (shows the dangers of responding when  you're on vacation and so I wasn't checking carefully enough).

I've done a bit of experimentation (now that I'm back at work) and I don't think closeResults() or ocnResetResults() are properly returning the memory to the pool (note, the memory may not be freed back to the OS, but at least Virtuoso should be able to re-use it for other waveform objects).

It's not clear from reading through the original request for closeResults() whether it was implemented with freeing memory in mind, although that seems to me a very good idea. It was mostly about closing any objects (and in particular file handles) so that psf directories could be deleted on disk.

So as I suggested, you should contact Cadence customer support about this (closeResults() should free memory used by any waveform objects which are no longer referenced via variables). I can raise it internally, but it's likely to get prioritised if coming from a real customer.

Regards,

Andrew.
Back to top
 
 
View Profile WWW   IP Logged
Andrew Beckett
Senior Fellow
******
Offline

Life, don't talk to
me about Life...

Posts: 1742
Bracknell, UK
Re: Virtuoso Slow Down after Large Memory exhaustion
Reply #12 - Feb 19th, 2018, 5:00am
 
A workaround seems to be to force a garbage collection after the call to closeResults() by calling gc() .

I had a discussion with R&D today (I filed an internal CCR 1881844 to suggest improving closeResults()). So it would be worth mentioning that if you log a case with customer support.

I'd be interested in hearing if calling gc() in your code improves matters.

Regards,

Andrew.
Back to top
 
 
View Profile WWW   IP Logged
Andrew Beckett
Senior Fellow
******
Offline

Life, don't talk to
me about Life...

Posts: 1742
Bracknell, UK
Re: Virtuoso Slow Down after Large Memory exhaustion
Reply #13 - Feb 19th, 2018, 10:33pm
 
It would also help to know how large the time domain data is in terms of the number of points. I suspect there must be a lot of points as otherwise 60,000 waveforms shouldn't be enough for excessive memory usage (I also would expect that number of waveforms to trigger garbage collection, probably). I'd like to try to reproduce some data with the characteristics of what you're seeing (logging a case with Cadence Customer Support would be best, but knowing this is better than nothing).

So after the line:

Code:
  selectResult("pss_td") 



can you add:

Code:
  printf("Number of points: %d\n" drVectorLength(drGetWaveformXVec(v("boke_1.Vb")))) 



That said, I have just realised (from looking at your script) that there's presumably a sweep around the pss in your setup because of the value() statements. So instead of above, I might need:

Code:
  printf("Sweep values: %L\n" sweepValues())
  sprintf(yyy, "boke_1.Vb", k)
  y = value( v(yyy), "Ain_dBV" Ain0_dBV )
  printf("Number of points: %d\n" drVectorLength(drGetWaveformXVec(y))) 



Thanks,

Andrew.
Back to top
 
 
View Profile WWW   IP Logged
cheap_salary
Senior Member
****
Offline



Posts: 162

Re: Virtuoso Slow Down after Large Memory exhaustion
Reply #14 - Feb 20th, 2018, 2:41am
 
Andrew Beckett wrote on Feb 19th, 2018, 10:33pm:
It would also help to know how large the time domain data is in terms of the number of points.
Same problem occurs even if I only use "pss_fd".
Actually, this thread starts for post processing of only "pss_fd".

Andrew Beckett wrote on Feb 19th, 2018, 5:00am:
I'd be interested in hearing if calling gc() in your code improves matters.
I have to get post processing results, so I use IC5 OCEAN for a while, since I can not get results at all by IC6 OCEAN.

Later I will confirm "gc()".


Back to top
 
 
View Profile   IP Logged
Pages: 1 2 
Send Topic Print
Copyright 2002-2024 Designer’s Guide Consulting, Inc. Designer’s Guide® is a registered trademark of Designer’s Guide Consulting, Inc. All rights reserved. Send comments or questions to editor@designers-guide.org. Consider submitting a paper or model.