You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-05 - 20:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Sun Jul 05, 2026 12:43:21)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
298348995951,5cyclictest401242-21grep11:00:011
298349995140,5cyclictest0-21swapper/211:00:402
298348995131,17cyclictest1-21systemd10:30:011
298348994923,23cyclictest325335-21idleruntime-cro08:40:001
298348994828,17cyclictest364830-21sendmail10:00:001
298348994828,17cyclictest364830-21sendmail10:00:001
298347994811,11cyclictest12650irq/32-eth012:15:370
298348994724,20cyclictest417854-21threads11:25:461
298348994722,22cyclictest390820-21unixbench_singl10:40:501
298347994732,12cyclictest3694966-21cups-browsed12:00:240
298347994732,12cyclictest3694966-21cups-browsed12:00:240
298349994635,7cyclictest1-21systemd10:25:012
298348994635,8cyclictest416445-21systemd-run11:25:021
29834899463,36cyclictest320250-21ntpq08:20:361
298348994524,17cyclictest0-21swapper/112:15:001
298348994524,17cyclictest0-21swapper/112:15:001
298348994522,20cyclictest350979-21switchtime09:35:471
29834899451,36cyclictest327753-21ntpq08:45:381
298349994436,5cyclictest422473-21cron11:35:002
298349994436,5cyclictest422473-21cron11:35:002
298349994436,5cyclictest308860-21systemd-run07:45:022
298349994435,6cyclictest352658-21basename09:40:012
29834999443,37cyclictest439754-21ssh12:02:032
29834999443,37cyclictest439754-21ssh12:02:032
298348994429,11cyclictest0-21swapper/112:20:011
29834899442,36cyclictest456881-21ntpq12:30:351
298348994423,18cyclictest0-21swapper/110:05:361
298348994421,20cyclictest0-21swapper/109:20:451
29834899441,12cyclictest309803-21ntpq07:45:401
298347994437,4cyclictest458914-21ssh12:35:010
298349994335,5cyclictest369499-21/usr/sbin/munin10:05:532
298349994335,5cyclictest331324-21idleruntime-cro09:00:012
298349994333,7cyclictest341206-21if_wlan009:20:292
298348994324,16cyclictest0-21swapper/110:50:271
29834899432,35cyclictest314228-21ntpq08:00:361
298348994321,17cyclictest0-21swapper/109:50:201
298348994321,17cyclictest0-21swapper/109:50:201
29834899431,6cyclictest317218-21ntpq08:10:361
29834899431,5cyclictest302268-21ntpq07:20:391
298347994335,5cyclictest317847-21systemd-run08:15:010
298348994236,3cyclictest0-21swapper/109:10:431
298348994234,5cyclictest410979-21fschecks_count11:15:291
29834899423,34cyclictest405307-21ssh11:05:391
29834899422,5cyclictest408374-21ntpq11:10:371
29834899421,6cyclictest315721-21ntpq08:05:341
29834899421,11cyclictest450851-21ntpq12:20:391
298347994233,6cyclictest417647-21wc11:25:420
29834899412,5cyclictest459935-21ntpq12:35:351
29834899412,5cyclictest333809-21ntpq09:05:421
29834899412,5cyclictest303764-21ntpq07:25:371
29834899412,5cyclictest303764-21ntpq07:25:371
29834899412,35cyclictest330682-21ntpq08:55:341
29834899412,34cyclictest326270-21ntpq08:40:391
298348994121,17cyclictest0-21swapper/110:30:481
29834899411,5cyclictest393238-21ntpq10:45:371
29834899411,4cyclictest347598-21ntpq09:30:381
29834899411,4cyclictest329236-21ntpq08:50:371
29834899411,36cyclictest321798-21ntpq08:25:391
29834899411,34cyclictest311203-21ntpq07:50:331
29834899411,12cyclictest318698-21ntpq08:15:351
29834899411,11cyclictest356832-21ntpq09:45:381
298347994133,5cyclictest420290-21sshd11:30:310
298347994133,5cyclictest420290-21sshd11:30:310
298349994033,3cyclictest0-21swapper/209:10:362
298349994032,5cyclictest444216-21aten_r2power_fr12:10:282
298349994032,5cyclictest444216-21aten_r2power_fr12:10:282
298349994032,5cyclictest419478-21systemd-run11:30:022
298349994032,5cyclictest302835-21cron07:25:012
298349994032,4cyclictest0-21swapper/209:25:402
29834899405,30cyclictest429595-21ssh11:45:351
29834899404,31cyclictest365924-21ssh10:00:371
298348994032,3cyclictest0-21swapper/111:30:361
298348994032,3cyclictest0-21swapper/111:30:361
29834899402,6cyclictest0-21swapper/110:20:341
29834899402,4cyclictest306749-21ntpq07:35:391
298348994023,14cyclictest0-21swapper/111:40:291
29834899401,9cyclictest299288-21ntp_states07:10:391
298348994017,20cyclictest0-21swapper/110:40:011
298348994015,22cyclictest345698-21ssh09:27:231
29834899401,4cyclictest308205-21ntpq07:40:391
29834899401,3cyclictest332261-21ntpq09:00:401
29834899401,14cyclictest422825-21ssh11:35:231
29834899401,11cyclictest338560-21ntpq09:15:411
29834899401,11cyclictest300793-21ntpq07:15:411
298349993932,4cyclictest314832-21cron08:05:012
298349993931,5cyclictest401242-21munin-plugin-st11:00:012
298349993930,6cyclictest421-21dbus-daemon11:10:392
298349993922,7cyclictest428-21mta-sts-daemon12:35:342
29834899392,5cyclictest305218-21ntpq07:30:361
29834899392,5cyclictest305218-21ntpq07:30:361
29834899392,33cyclictest453901-21ntpq12:25:381
29834899392,31cyclictest371539-21diskstats10:10:281
29834899391,9cyclictest313274-21users07:55:471
29834899391,5cyclictest323250-21ntpq08:30:381
29834899391,5cyclictest323250-21ntpq08:30:381
29834899391,12cyclictest374999-21ssh10:15:351
298347993932,4cyclictest446864-21latency_hist12:15:020
298347993932,4cyclictest446864-21latency_hist12:15:020
298349993830,3cyclictest0-21swapper/210:30:412
298349993828,7cyclictest1-21systemd11:25:022
298349993828,6cyclictest421-21dbus-daemon12:00:022
298349993820,10cyclictest337766-21head09:15:252
298348993830,4cyclictest0-21swapper/109:40:371
29834899382,11cyclictest441737-21modprobe12:05:341
29834899382,11cyclictest441737-21modprobe12:05:341
29834899381,9cyclictest433508-21ssh11:51:321
29834899381,5cyclictest435769-21ntpq11:55:421
29834899381,12cyclictest403069-21ssh11:01:081
29834899381,10cyclictest438898-21ps12:00:431
29834899381,10cyclictest438898-21ps12:00:431
298347993831,5cyclictest342122-21unixbench_singl09:20:470
298347993831,4cyclictest402809-21timerwakeupswit11:00:500
29834999378,8cyclictest36-21ksoftirqd/209:45:422
298349993728,5cyclictest426480-21missed_timers11:40:312
298347993727,7cyclictest346700-21latency_hist09:30:030
298347993718,11cyclictest1003-21mta-sts-daemon08:50:310
298349993628,5cyclictest413394-21sendmail11:20:012
298349993627,5cyclictest405678-21switchtime11:05:512
298347993629,4cyclictest151ktimers/010:10:350
298347993628,5cyclictest399114-21/usr/sbin/munin10:55:360
298349993526,6cyclictest1000-21mta-sts-daemon07:25:392
298349993526,6cyclictest1000-21mta-sts-daemon07:25:392
298349993523,3cyclictest171rcu_preempt09:05:262
298347993531,2cyclictest0-21swapper/010:35:010
298347993529,4cyclictest450275-21aten_r2power_fr12:20:260
298347993529,4cyclictest368544-21fschecks_count10:05:300
298347993526,4cyclictest0-21swapper/008:30:360
298347993526,4cyclictest0-21swapper/008:30:360
29834799352,28cyclictest335378-21ssh09:10:360
298349993427,4cyclictest458731-21kworker/u8:2-xprtiod12:35:012
298349993427,3cyclictest0-21swapper/210:40:452
298349993424,7cyclictest452832-21munin-run12:25:012
298349993424,3cyclictest171rcu_preempt10:15:342
298349993410,6cyclictest36-21ksoftirqd/212:10:012
298349993410,6cyclictest36-21ksoftirqd/212:10:012
29834899341,3cyclictest298359-21munin-run07:10:011
29834799349,9cyclictest432699-21ssh11:50:410
298347993430,2cyclictest0-21swapper/010:00:000
298347993430,2cyclictest0-21swapper/010:00:000
298347993427,4cyclictest375509-21unixbench-2d10:15:460
298347993421,7cyclictest12650irq/32-eth012:35:410
298347993418,10cyclictest12650irq/32-eth009:30:410
298349993327,3cyclictest0-21swapper/207:55:012
298349993324,6cyclictest454330-21/usr/sbin/munin12:25:492
298349993319,4cyclictest57-21kcompactd009:55:372
298349993319,4cyclictest57-21kcompactd009:55:372
298349993314,6cyclictest3694966-21cups-browsed10:51:352
298347993328,3cyclictest358762-21munin-run09:50:010
298347993326,4cyclictest301352-21sendmail07:20:010
298347993325,6cyclictest352667-21idleruntime-cro09:40:010
29834999327,10cyclictest351ktimers/211:40:022
298349993225,4cyclictest434805-21latency_hist11:55:022
298349993225,3cyclictest0-21swapper/209:45:012
298349993224,6cyclictest346841-21sh09:30:162
298349993224,5cyclictest389208-21latency_hist10:40:022
298349993223,6cyclictest421-21dbus-daemon11:50:012
298349993222,8cyclictest421-21dbus-daemon09:55:022
298349993222,8cyclictest421-21dbus-daemon09:55:022
298349993222,7cyclictest318522-21latency08:15:312
298349993222,5cyclictest351ktimers/207:15:352
298349993219,2cyclictest171rcu_preempt07:30:352
298349993219,2cyclictest171rcu_preempt07:30:352
298349993217,3cyclictest171rcu_preempt07:35:452
298349993214,8cyclictest171rcu_preempt10:11:432
298347993226,3cyclictest191rcuc/008:50:020
298347993225,5cyclictest421-21dbus-daemon10:40:010
298347993222,4cyclictest151ktimers/010:40:450
298347993217,5cyclictest171rcu_preempt08:35:470
298347993216,3cyclictest171rcu_preempt10:45:440
298347993215,4cyclictest1002-21mta-sts-daemon07:30:400
298347993215,4cyclictest1002-21mta-sts-daemon07:30:400
298349993124,4cyclictest313316-21munin-run08:00:012
298349993121,6cyclictest0-21swapper/210:45:332
298349993115,3cyclictest171rcu_preempt10:26:412
29834799317,7cyclictest14-21ksoftirqd/009:40:460
298347993125,4cyclictest298346-21cyclictest10:30:010
298347993124,4cyclictest442319-21sed12:05:470
298347993124,4cyclictest442319-21sed12:05:470
298347993114,4cyclictest0-21swapper/010:20:270
298347993111,7cyclictest151ktimers/011:46:040
29834999307,6cyclictest36-21ksoftirqd/210:00:472
298349993024,3cyclictest0-21swapper/208:30:312
298349993024,3cyclictest0-21swapper/208:30:312
298349993017,3cyclictest171rcu_preempt08:05:352
298349993016,3cyclictest171rcu_preempt07:10:422
29834799307,12cyclictest151ktimers/011:10:360
298347993024,3cyclictest411937-21unixbench_multi11:15:480
298347993022,5cyclictest414039-21grep11:20:290
298347993016,2cyclictest171rcu_preempt09:54:590
298347993016,2cyclictest171rcu_preempt09:54:590
298347993014,5cyclictest331298-21sshd08:59:310
298349992923,4cyclictest298346-21cyclictest08:40:002
298349992922,4cyclictest448450-21unixbench_singl12:15:512
298349992921,5cyclictest328834-21fschecks_count08:50:292
298349992921,5cyclictest326684-21unixbench-2d08:40:482
298349992920,3cyclictest171rcu_preempt09:00:292
298349992919,5cyclictest171rcu_preempt08:25:352
298349992919,3cyclictest171rcu_preempt07:45:272
298349992915,2cyclictest171rcu_preempt08:20:322
298349992912,3cyclictest171rcu_preempt07:10:022
29834799298,5cyclictest151ktimers/011:55:400
29834799297,7cyclictest14-21ksoftirqd/009:15:380
29834799296,21cyclictest151ktimers/007:35:310
29834799295,16cyclictest428-21mta-sts-daemon07:25:490
29834799295,16cyclictest428-21mta-sts-daemon07:25:490
29834799294,16cyclictest420-21avahi-daemon11:35:370
298347992923,4cyclictest311825-21grep07:55:010
298347992923,3cyclictest191rcuc/008:42:490
298347992923,3cyclictest0-21swapper/008:05:010
298347992922,4cyclictest151ktimers/007:40:430
298347992922,4cyclictest0-21swapper/008:05:350
298347992921,4cyclictest151ktimers/011:44:130
298347992917,9cyclictest0-21swapper/010:50:300
298347992916,4cyclictest171rcu_preempt10:02:000
298347992916,3cyclictest171rcu_preempt08:30:010
29834799291,24cyclictest3694966-21cups-browsed09:00:360
298347992912,4cyclictest151ktimers/011:05:200
298347992912,3cyclictest171rcu_preempt12:28:470
298347992911,15cyclictest191rcuc/007:45:330
298349992816,2cyclictest171rcu_preempt08:10:372
298349992815,6cyclictest171rcu_preempt08:45:342
29834799286,11cyclictest151ktimers/009:05:400
29834799280,17cyclictest312700-21ld07:55:350
29834799278,16cyclictest151ktimers/008:15:360
298347992722,3cyclictest151ktimers/007:10:400
298347992719,5cyclictest151ktimers/007:20:420
29834799268,3cyclictest171rcu_preempt08:20:260
29834799233,6cyclictest14-21ksoftirqd/007:10:020
29835099212,8cyclictest0-21swapper/312:02:033
29835099212,8cyclictest0-21swapper/312:02:033
29835099192,7cyclictest0-21swapper/311:50:373
29835099192,6cyclictest0-21swapper/310:50:383
29835099182,7cyclictest0-21swapper/311:20:413
29835099182,7cyclictest0-21swapper/309:10:373
29835099182,6cyclictest0-21swapper/308:50:353
29835099181,7cyclictest0-21swapper/310:55:393
29835099171,9cyclictest0-21swapper/309:20:363
29835099163,5cyclictest0-21swapper/308:00:373
29835099162,8cyclictest0-21swapper/311:00:453
29835099162,7cyclictest0-21swapper/310:30:343
29835099162,6cyclictest0-21swapper/307:15:343
29835099161,9cyclictest0-21swapper/311:05:443
29835099153,8cyclictest0-21swapper/311:40:373
29835099151,8cyclictest0-21swapper/309:55:373
29835099151,8cyclictest0-21swapper/309:55:373
29835099151,7cyclictest0-21swapper/308:10:373
29835099151,7cyclictest0-21swapper/307:40:413
29835099143,7cyclictest0-21swapper/311:10:423
29835099142,7cyclictest0-21swapper/312:15:423
29835099142,7cyclictest0-21swapper/310:25:413
29835099142,7cyclictest0-21swapper/309:30:403
29835099142,7cyclictest0-21swapper/309:00:363
29835099142,6cyclictest0-21swapper/310:20:343
29835099142,6cyclictest0-21swapper/307:35:363
29835099135,5cyclictest0-21swapper/312:05:343
29835099135,5cyclictest0-21swapper/312:05:343
29835099133,7cyclictest0-21swapper/310:35:383
29835099133,6cyclictest0-21swapper/309:05:373
29835099133,5cyclictest0-21swapper/309:40:343
29835099133,4cyclictest0-21swapper/308:15:353
29835099132,7cyclictest0-21swapper/312:25:323
29835099132,7cyclictest0-21swapper/312:25:013
29835099132,7cyclictest0-21swapper/311:15:313
29835099132,7cyclictest0-21swapper/309:50:403
29835099132,7cyclictest0-21swapper/309:50:403
29835099132,7cyclictest0-21swapper/309:35:263
29835099132,7cyclictest0-21swapper/309:25:403
29835099132,6cyclictest0-21swapper/310:15:353
29835099132,6cyclictest0-21swapper/308:30:353
29835099132,6cyclictest0-21swapper/308:30:353
29835099131,8cyclictest0-21swapper/312:10:383
29835099131,8cyclictest0-21swapper/312:10:383
29835099131,7cyclictest0-21swapper/311:30:313
29835099131,7cyclictest0-21swapper/311:30:313
29835099131,6cyclictest0-21swapper/310:40:363
29835099125,4cyclictest431ktimers/307:10:013
29835099124,5cyclictest0-21swapper/311:45:393
29835099124,5cyclictest0-21swapper/308:20:373
29835099123,6cyclictest0-21swapper/311:25:373
29835099123,6cyclictest0-21swapper/310:45:383
29835099123,6cyclictest0-21swapper/310:05:413
29835099123,6cyclictest0-21swapper/310:00:363
29835099123,6cyclictest0-21swapper/309:15:363
29835099123,6cyclictest0-21swapper/307:30:353
29835099123,6cyclictest0-21swapper/307:30:353
29835099123,4cyclictest0-21swapper/308:55:343
29835099122,7cyclictest0-21swapper/310:10:393
29835099122,6cyclictest0-21swapper/311:55:443
29835099122,6cyclictest0-21swapper/307:50:353
29835099122,5cyclictest0-21swapper/309:45:383
29835099121,7cyclictest0-21swapper/311:35:303
29835099121,7cyclictest0-21swapper/308:40:423
29835099121,6cyclictest0-21swapper/312:30:303
29835099121,6cyclictest0-21swapper/308:05:353
29835099114,5cyclictest0-21swapper/312:35:293
29835099112,5cyclictest0-21swapper/308:25:263
29835099112,5cyclictest0-21swapper/307:45:353
29835099112,5cyclictest0-21swapper/307:25:353
29835099112,5cyclictest0-21swapper/307:25:353
29835099111,7cyclictest0-21swapper/307:20:313
29835099111,6cyclictest0-21swapper/308:35:363
29835099111,6cyclictest0-21swapper/307:55:333
29835099102,4cyclictest0-21swapper/307:10:273
2983509991,5cyclictest0-21swapper/308:45:263
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional