You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-09 - 12:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot4s.osadl.org (updated Tue Dec 09, 2025 00:45:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
6175996126,23cyclictest21-21ksoftirqd/120:20:121
617599482,39cyclictest8054-21ssh00:35:121
6161994423,7cyclictest12-21ksoftirqd/023:40:120
617599414,25cyclictest22257-21ssh23:05:141
6175993918,13cyclictest629-21Async14:00:481
6195993616,13cyclictest29566irq/58-eth022:40:143
618499342,16cyclictest26676-21/usr/sbin/munin21:50:282
616199342,4cyclictest131rcu_preempt00:20:130
616199342,4cyclictest131rcu_preempt00:20:120
617599332,25cyclictest9247-21ssh21:20:151
617599332,23cyclictest22181-21mailstats21:45:291
616199334,26cyclictest28778-21ssh22:30:140
6195993126,3cyclictest29566irq/58-eth022:35:143
6184993117,8cyclictest6147-21cyclictest22:30:142
616199311,27cyclictest31466-21ssh23:30:140
617599302,22cyclictest11280-21ssh22:19:291
616199301,22cyclictest634-21ssh00:10:150
616199300,23cyclictest0-21swapper/021:55:130
618499291,21cyclictest29743-21ssh22:50:162
618499291,18cyclictest629-21Async14:00:482
618499291,16cyclictest838-21ssh21:15:152
617599293,19cyclictest31446-21ssh23:30:131
617599292,20cyclictest23447-21/usr/sbin/munin22:05:291
616199293,23cyclictest24920-21ssh00:25:160
616199293,21cyclictest21425-21sh22:45:140
6195992815,9cyclictest29566irq/58-eth023:40:143
618499283,10cyclictest0-21swapper/222:23:292
618499281,20cyclictest13113-21sh22:40:132
618499281,17cyclictest24127-21ssh23:45:142
617599283,18cyclictest5532-21ssh22:55:151
617599281,19cyclictest30585-21ssh23:10:151
616199282,20cyclictest8069-21ssh00:35:150
617599272,19cyclictest6772-21sh21:55:281
616199271,23cyclictest23020-21ssh23:25:160
618499264,14cyclictest14798-21ssh23:20:152
6184992612,11cyclictest9275-21wc00:15:192
618499261,10cyclictest0-21swapper/223:55:142
616199263,18cyclictest7483-21ssh23:35:140
616199263,14cyclictest0-21swapper/000:30:150
616199262,21cyclictest25849-21ssh21:30:140
6195992517,5cyclictest29566irq/58-eth022:30:143
6195992516,6cyclictest29566irq/58-eth023:25:163
6184992512,6cyclictest24587-21diskmemload23:25:192
617599251,2cyclictest0-21swapper/119:15:171
617599251,19cyclictest27763-21ssh22:10:141
6161992518,5cyclictest12-21ksoftirqd/022:55:160
618499242,8cyclictest20791-21ssh23:03:282
618499242,8cyclictest20791-21ssh23:03:272
618499241,9cyclictest18357-21ssh21:45:152
618499241,8cyclictest9957-21ssh22:17:312
6175992413,7cyclictest16685-21cron00:20:011
616199248,11cyclictest0-21swapper/022:50:170
616199242,19cyclictest16024-21timerandwakeup22:20:290
619599234,14cyclictest0-21swapper/322:50:173
6195992312,5cyclictest0-21swapper/323:10:143
618499239,11cyclictest24223-21users22:05:302
618499233,15cyclictest0-21swapper/222:04:292
618499232,16cyclictest0-21swapper/223:15:282
618499232,16cyclictest0-21swapper/221:58:272
618499232,10cyclictest24587-21diskmemload22:35:162
6184992312,8cyclictest11950-21usage_EthcTask23:35:302
618499231,19cyclictest30639-21grep19:35:162
617599231,19cyclictest26587-21ps21:30:191
616199232,18cyclictest15107-21sed22:00:290
6161992312,8cyclictest27365-21cron22:10:000
616199231,18cyclictest997-21cut21:15:170
619599223,13cyclictest0-21swapper/300:25:153
619599221,15cyclictest0-21swapper/300:30:153
618499229,10cyclictest4359-21sshd00:10:292
618499227,7cyclictest22203-21tr22:25:222
618499223,10cyclictest22878-21kthreadcore20:00:212
618499222,15cyclictest0-21swapper/223:09:272
618499222,15cyclictest0-21swapper/222:11:282
618499221,3cyclictest131rcu_preempt21:32:292
618499221,18cyclictest32673-21ssh00:30:142
618499221,17cyclictest11918-21kthreadcore20:55:242
618499221,14cyclictest73-21kswapd023:40:152
618499221,13cyclictest13617-21ssh21:40:252
618499221,13cyclictest0-21swapper/200:00:222
617599222,14cyclictest10178-21ntp_states23:15:291
617599222,11cyclictest17033-21apt-get00:20:131
617599222,11cyclictest17033-21apt-get00:20:121
616199226,11cyclictest629-21Async14:00:480
616199225,11cyclictest12-21ksoftirqd/023:20:140
616199222,15cyclictest16778-21ntp_states22:40:270
6161992211,8cyclictest31274-21missed_timers22:10:240
6195992115,4cyclictest29666irq/59-eth023:17:023
6195992115,4cyclictest29666irq/59-eth021:17:303
6195992111,7cyclictest0-21swapper/323:20:313
618499216,7cyclictest370-21kworker/u8:022:45:152
618499213,4cyclictest131rcu_preempt19:20:012
618499213,15cyclictest17838-21sed21:25:172
618499212,11cyclictest28509-21ntp_states00:05:272
618499211,9cyclictest5713-21fschecks_count22:55:192
618499211,8cyclictest91-21pr/ttyS023:55:012
618499211,6cyclictest24587-21diskmemload00:36:052
618499211,4cyclictest24587-21diskmemload23:10:162
618499211,16cyclictest0-21swapper/219:25:132
618499211,12cyclictest23843-21tr20:35:182
618499211,12cyclictest17045-21ssh00:20:142
618499211,12cyclictest17045-21ssh00:20:132
618499211,11cyclictest5789-21ssh21:35:312
618499211,10cyclictest8947-21latency_hist20:20:012
618499211,10cyclictest12380-21kthreadcore21:20:252
6184992110,9cyclictest0-21swapper/200:25:202
617599214,11cyclictest435-21ssh21:52:291
6175992113,5cyclictest0-21swapper/120:40:271
6175992112,6cyclictest7870-21ssh21:37:281
6175992112,6cyclictest6274-21ssh22:35:221
6175992112,6cyclictest0-21swapper/120:50:231
617599211,15cyclictest25520-21ls19:30:021
617599211,13cyclictest10039-21ssh21:40:121
617599210,18cyclictest16982-21ls22:40:291
616199212,13cyclictest28328-21apt-get22:30:010
616199212,13cyclictest12-21ksoftirqd/020:55:170
6161992116,3cyclictest0-21swapper/021:25:140
6161992114,4cyclictest0-21swapper/020:15:180
6161992113,6cyclictest10927-21sshd19:12:170
6161992112,7cyclictest24577-21cron21:10:000
6161992111,7cyclictest0-21swapper/022:17:270
616199211,16cyclictest4756-21rm22:35:150
616199211,15cyclictest30351-21ssh21:50:260
616199211,15cyclictest1933-21cut21:35:170
6195992015,3cyclictest29666irq/59-eth021:27:313
6195992014,3cyclictest0-21swapper/300:20:203
6195992014,3cyclictest0-21swapper/300:20:203
6195992013,4cyclictest0-21swapper/323:05:153
619599201,16cyclictest29566irq/58-eth023:35:133
618499206,11cyclictest0-21swapper/220:30:212
618499206,11cyclictest0-21swapper/220:25:182
618499201,9cyclictest6770-21grep19:40:302
618499201,9cyclictest3773-21kthreadcore20:45:232
618499201,9cyclictest25806-21fschecks_time19:30:182
618499201,9cyclictest15316-21fschecks_count21:00:192
618499201,9cyclictest10936-21cron19:15:022
618499201,3cyclictest0-21swapper/223:30:082
618499201,16cyclictest27567-21tr21:10:252
618499201,15cyclictest5698-21cut20:50:172
618499201,15cyclictest17511-21fschecks_time19:55:172
618499201,15cyclictest16204-21grep19:20:162
618499201,10cyclictest9539-21irqrtprio20:20:192
618499201,10cyclictest31872-21wc20:40:272
618499201,10cyclictest13039-21irqrtprio19:50:192
617599209,9cyclictest1208-21awk22:30:351
617599207,4cyclictest131rcu_preempt20:30:021
617599203,13cyclictest24829-21ssh00:05:151
617599203,13cyclictest1-21systemd20:15:121
6175992012,6cyclictest0-21swapper/123:40:191
6175992011,6cyclictest0-21swapper/123:30:011
617599201,15cyclictest32403-21ssh23:50:151
616199209,8cyclictest29163-21cat00:05:320
616199209,3cyclictest131rcu_preempt23:10:310
616199205,12cyclictest22860-21kthreadcore20:00:210
616199204,10cyclictest0-21swapper/019:30:150
616199203,14cyclictest21667-21tr21:45:230
616199202,14cyclictest12939-21ntp_states21:20:290
616199202,13cyclictest22384-21diskstats23:05:170
616199201,9cyclictest1624966ssh00:19:140
616199201,9cyclictest13853-21/usr/sbin/munin23:00:160
616199201,9cyclictest13853-21/usr/sbin/munin23:00:160
6161992013,5cyclictest11972-21open_inodes23:55:280
616199201,15cyclictest17355-21apt-get19:55:120
616199201,14cyclictest19949-21tr00:00:240
616199201,13cyclictest17128-21latency_hist19:55:010
619599192,14cyclictest29566irq/58-eth021:55:133
619599192,12cyclictest0-21swapper/300:35:153
6195991911,5cyclictest29566irq/58-eth000:10:143
619599191,12cyclictest0-21swapper/321:45:163
619599191,12cyclictest0-21swapper/320:50:293
6195991910,6cyclictest0-21swapper/322:57:293
618499192,12cyclictest0-21swapper/221:05:142
618499192,10cyclictest32435-21kthreadcore20:10:212
618499192,10cyclictest32435-21kthreadcore20:10:212
618499190,16cyclictest8826-21kthreadcore19:45:202
617599198,8cyclictest19934-21cut00:00:231
617599191,7cyclictest26306-21kthreadcore21:10:221
6175991913,4cyclictest0-21swapper/121:00:241
6175991912,5cyclictest0-21swapper/120:55:271
6175991912,5cyclictest0-21swapper/120:35:171
6175991912,4cyclictest0-21swapper/120:00:271
6175991911,6cyclictest22609-21ssh23:24:591
6175991911,6cyclictest18432-21ssh22:04:081
6175991911,6cyclictest0-21swapper/121:19:281
6175991911,5cyclictest0-21swapper/119:10:181
617599191,12cyclictest12702-21grep19:50:161
617599191,10cyclictest17145-21tr19:20:191
6175991910,7cyclictest11157-21ntp_states23:35:261
6175991910,6cyclictest0-21swapper/120:45:201
617599190,17cyclictest6186-21ls19:40:231
616199199,8cyclictest28349-21apt-get20:40:130
616199195,10cyclictest23203-21apt-get20:35:010
616199193,13cyclictest27345-21kthreadcore21:10:250
6161991911,6cyclictest25399-21sshd19:27:170
616199191,14cyclictest25971-21cut23:45:210
6161991910,7cyclictest7008-21cut23:15:200
619599189,6cyclictest29566irq/58-eth023:50:153
619599189,6cyclictest0-21swapper/321:54:463
619599189,6cyclictest0-21swapper/321:11:163
619599189,5cyclictest29566irq/58-eth022:25:143
619599189,5cyclictest29566irq/58-eth000:15:163
619599183,11cyclictest0-21swapper/322:00:283
6195991812,3cyclictest29566irq/58-eth023:30:163
6195991811,4cyclictest0-21swapper/322:45:133
619599181,12cyclictest0-21swapper/321:40:023
6195991810,6cyclictest0-21swapper/300:03:183
618499181,8cyclictest6214-21latency_hist19:10:002
617599189,7cyclictest2637-21cron19:40:021
617599186,9cyclictest24077-21ssh22:25:261
6175991812,4cyclictest0-21swapper/120:10:191
6175991812,4cyclictest0-21swapper/120:10:181
6175991811,5cyclictest5726-21ssh00:10:531
6175991811,5cyclictest3178-21ssh22:52:101
6175991811,5cyclictest28150-21ssh22:48:131
6175991811,5cyclictest0-21swapper/119:45:181
617599181,13cyclictest21225-21ntpq21:25:271
617599181,13cyclictest15834-21awk22:20:271
617599181,13cyclictest14455-21sed23:00:181
617599181,13cyclictest14455-21sed23:00:171
616199187,8cyclictest12-21ksoftirqd/020:45:230
616199182,13cyclictest32747-21/usr/sbin/munin23:50:200
616199182,13cyclictest11063-21tr20:20:200
6161991812,4cyclictest8042-21tail19:45:170
616199181,14cyclictest0-21swapper/020:10:150
616199181,14cyclictest0-21swapper/020:10:140
616199181,13cyclictest28126-21grep20:40:010
616199181,13cyclictest11065-21ssh21:40:210
619599179,6cyclictest0-21swapper/323:02:183
619599179,6cyclictest0-21swapper/323:02:173
619599179,6cyclictest0-21swapper/300:08:223
619599178,6cyclictest29566irq/58-eth021:20:053
619599172,9cyclictest0-21swapper/319:30:153
619599172,10cyclictest29566irq/58-eth021:30:143
6195991711,4cyclictest0-21swapper/323:55:213
619599171,10cyclictest0-21swapper/319:15:183
6195991710,4cyclictest0-21swapper/323:45:163
6195991710,3cyclictest0-21swapper/321:40:253
617599179,6cyclictest13752-21ssh23:56:311
6175991711,4cyclictest0-21swapper/120:30:231
6175991711,4cyclictest0-21swapper/120:05:301
6175991711,4cyclictest0-21swapper/100:25:291
6175991710,5cyclictest324-21ssh00:30:181
6175991710,4cyclictest0-21swapper/121:05:191
616199174,10cyclictest16910-21kthreadcore20:25:240
616199172,12cyclictest18126-21cut19:20:210
616199171,13cyclictest6207-21cat19:10:000
616199171,13cyclictest3408-21sed19:40:190
616199171,13cyclictest1909-21grep19:35:280
616199171,12cyclictest5510-21acpi20:50:130
616199171,12cyclictest27691-21tr20:05:200
616199171,12cyclictest15952-21grep19:20:010
619599168,5cyclictest29566irq/58-eth022:12:063
619599167,6cyclictest0-21swapper/322:15:563
619599166,7cyclictest0-21swapper/322:25:023
619599162,10cyclictest0-21swapper/321:00:173
619599162,10cyclictest0-21swapper/320:55:183
619599162,10cyclictest0-21swapper/319:25:133
619599162,10cyclictest0-21swapper/319:20:143
6195991611,3cyclictest0-21swapper/322:05:313
617599161,6cyclictest28821-21kthreadcore19:30:231
6175991610,4cyclictest20494-21cut19:55:231
619599154,4cyclictest0-21swapper/321:05:143
619599152,9cyclictest0-21swapper/320:10:153
619599152,9cyclictest0-21swapper/320:10:143
619599149,3cyclictest0-21swapper/319:51:223
619599144,6cyclictest0-21swapper/319:40:143
619599141,10cyclictest0-21swapper/320:35:213
617599147,5cyclictest6301-21sed19:10:021
619599138,3cyclictest0-21swapper/320:40:233
619599138,3cyclictest0-21swapper/320:25:293
619599138,3cyclictest0-21swapper/320:20:163
619599131,8cyclictest29566irq/58-eth019:45:133
619599127,3cyclictest0-21swapper/319:12:173
619599124,6cyclictest0-21swapper/320:00:183
619599122,7cyclictest0-21swapper/320:20:013
619599121,9cyclictest0-21swapper/319:55:193
619599121,7cyclictest0-21swapper/320:50:013
619599121,7cyclictest0-21swapper/319:35:203
619599112,7cyclictest0-21swapper/320:05:163
619599111,7cyclictest0-21swapper/320:35:003
619599111,7cyclictest0-21swapper/319:10:003
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional