You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-03 - 03:32

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Wed Jun 03, 2026 00:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
18308210190,6sleep00-21swapper/019:09:580
1802729468,8sleep20-21swapper/219:06:232
1800829364,9sleep30-21swapper/319:06:083
1817728366,13sleep10-21swapper/119:08:181
18403992727,0cyclictest0-21swapper/321:37:303
1840399270,26cyclictest0-21swapper/323:47:423
1839899262,1cyclictest6140-21diskmemload00:23:591
1839899261,19cyclictest6140-21diskmemload21:13:391
1839499250,24cyclictest0-21swapper/021:59:050
265062230,0sleep00-21swapper/023:55:190
18400992221,1cyclictest32-21ksoftirqd/222:59:052
1840099220,20cyclictest0-21swapper/223:23:142
18400992121,0cyclictest0-21swapper/220:49:282
1840099210,21cyclictest0-21swapper/223:16:242
1840099210,1cyclictest0-21swapper/223:38:032
1839899212,15cyclictest11793-21ntp_states22:10:211
1839899210,1cyclictest0-21swapper/122:06:111
1839899210,18cyclictest3240-21/usr/sbin/munin22:25:151
1839899210,0cyclictest0-21swapper/122:45:171
51842200,0sleep00-21swapper/000:01:530
1840399200,1cyclictest0-21swapper/322:40:373
1840399200,19cyclictest0-21swapper/323:08:153
1840099200,20cyclictest0-21swapper/200:00:342
1840099190,17cyclictest0-21swapper/221:18:152
1839899196,1cyclictest27984-21ssh21:41:211
1839899195,9cyclictest8365-21/usr/sbin/munin23:25:191
1839899193,15cyclictest6140-21diskmemload21:50:141
1839499199,4cyclictest25030-21cpuspeed_turbos21:40:140
1839499195,8cyclictest24299-21perf21:40:010
1839499190,18cyclictest0-21swapper/021:24:520
1840399183,14cyclictest0-21swapper/322:30:133
18398991817,0cyclictest0-21swapper/121:37:271
1839899180,1cyclictest0-21swapper/100:00:271
18394991810,6cyclictest14301-21apache200:35:190
18394991810,6cyclictest14301-21apache200:35:190
1839499180,17cyclictest0-21swapper/023:13:050
18403991716,1cyclictest0-21swapper/323:24:553
1840099170,17cyclictest0-21swapper/200:10:192
1839899175,10cyclictest4265-21turbostat22:45:001
18398991716,1cyclictest0-21swapper/100:11:171
1839499170,17cyclictest0-21swapper/022:49:390
1839499170,17cyclictest0-21swapper/022:31:420
18403991616,0cyclictest0-21swapper/322:36:183
1840099160,3cyclictest0-21swapper/222:45:532
1839899160,2cyclictest0-21swapper/123:46:481
1839899160,16cyclictest0-21swapper/123:31:151
258602150,0sleep00-21swapper/022:19:180
18403991514,1cyclictest0-21swapper/322:50:223
18403991514,1cyclictest0-21swapper/322:50:223
18403991512,2cyclictest15922-21mailstats23:10:223
1840399150,3cyclictest0-21swapper/300:37:313
1840399150,3cyclictest0-21swapper/300:37:303
1840399150,1cyclictest0-21swapper/322:07:583
1840399150,1cyclictest0-21swapper/300:28:553
1840099151,1cyclictest141rcu_preempt22:02:012
1840099150,15cyclictest0-21swapper/221:46:352
1840099150,15cyclictest0-21swapper/221:43:582
1839899153,2cyclictest225-21systemd-journal22:35:001
1839899150,2cyclictest0-21swapper/123:00:361
18403991414,0cyclictest0-21swapper/321:50:323
18403991413,1cyclictest0-21swapper/323:04:063
1840399141,13cyclictest0-21swapper/322:24:293
18403991411,2cyclictest32633-21cron21:45:003
18403991410,3cyclictest9504-21cron20:10:003
18403991410,3cyclictest8529-21tr21:30:293
1840099148,5cyclictest55850irq/129-enp2s0-23:14:472
1840099145,7cyclictest27718-21rm00:35:002
1840099145,6cyclictest3417-21ssh22:25:132
18400991414,0cyclictest0-21swapper/222:15:052
18400991410,3cyclictest4215-21wc00:20:212
18400991410,2cyclictest10152-21grep00:05:192
1839899146,7cyclictest38150irq/131-snd_hda23:06:081
1839899142,11cyclictest31562-21taskset00:17:251
18398991412,1cyclictest0-21swapper/100:31:061
18398991411,2cyclictest24386-21munin-run19:30:011
18398991410,3cyclictest16643-21cron20:25:001
1839899140,12cyclictest21749-21grep19:20:101
18394991410,2cyclictest4616-21open_files22:25:200
1839499140,8cyclictest1518-21basename22:05:000
1839499140,4cyclictest0-21swapper/000:10:030
1839499140,14cyclictest0-21swapper/022:22:140
1840399139,3cyclictest3318-21grep19:55:183
1840399139,3cyclictest27744-21cut00:15:193
1840399139,3cyclictest23901-21cat23:15:183
1840399139,3cyclictest17808-21head23:50:183
18403991313,0cyclictest0-21swapper/322:11:043
18403991310,2cyclictest31511-21grep21:25:013
18403991310,2cyclictest28531-21sed20:45:283
18403991310,2cyclictest27010-21wc22:20:003
18403991310,2cyclictest15236-21grep20:20:163
1840099133,9cyclictest18519-21/usr/sbin/munin21:55:172
1840099131,2cyclictest3522-21ssh22:44:262
18400991310,2cyclictest2030-21head23:40:262
18400991310,2cyclictest18557-21wc20:25:252
18400991310,2cyclictest15203-21ssh22:50:272
18400991310,2cyclictest15203-21ssh22:50:272
18400991310,2cyclictest13179-21expr00:25:262
1840099130,13cyclictest0-21swapper/222:10:432
1839899138,4cyclictest26277-21taskset22:38:281
18398991310,2cyclictest30580-21grep19:45:151
1839899130,4cyclictest225-21systemd-journal19:20:001
1839899130,1cyclictest0-21swapper/123:37:561
1839899130,1cyclictest0-21swapper/121:21:571
1839899130,12cyclictest18385-21sh21:55:161
1839899130,12cyclictest0-21swapper/122:16:011
1839499139,3cyclictest2827-21grep22:05:170
1839499137,4cyclictest27389-21grep19:40:000
1839499135,4cyclictest21388-21perf19:20:000
1839499131,6cyclictest28989-21munin-run20:50:000
18394991310,2cyclictest6819-21head23:05:190
18394991310,2cyclictest3202-21grep21:00:270
18394991310,2cyclictest23869-21wc19:25:000
18394991310,2cyclictest16631-21wc23:49:590
18394991310,2cyclictest11363-21sed22:10:170
1840399129,2cyclictest899-21cat00:00:003
1840399129,2cyclictest3305-21sed21:00:273
1840399129,2cyclictest32676-21head21:25:173
1840399129,2cyclictest21681-21grep19:20:093
1840399129,2cyclictest19882-21grep00:10:213
1840399129,2cyclictest1401-21sh00:00:143
1840399128,3cyclictest20299-21grep20:30:183
1840399127,2cyclictest25215-21grep20:40:173
1840399126,2cyclictest225-21systemd-journal00:35:003
18403991211,1cyclictest0-21swapper/323:33:183
1840399120,2cyclictest0-21swapper/300:05:183
1840399120,11cyclictest0-21swapper/322:03:313
1840399120,10cyclictest0-21swapper/321:45:123
1840099129,2cyclictest9353-21grep23:45:162
1840099129,2cyclictest8469-21tr21:30:292
1840099129,2cyclictest3291-21grep19:55:182
1840099129,2cyclictest2846-21grep22:05:172
1840099129,2cyclictest18902-21grep23:50:272
1840099129,2cyclictest1316-21grep21:25:262
1840099123,5cyclictest225-21systemd-journal21:55:002
1840099123,5cyclictest225-21systemd-journal19:42:002
1840099121,10cyclictest0-21swapper/200:39:322
1840099121,10cyclictest0-21swapper/200:39:322
18400991210,1cyclictest28084-21cut22:20:172
1840099120,1cyclictest0-21swapper/223:25:132
1839899129,2cyclictest9810-21cut20:10:131
1839899129,2cyclictest9512-21apt20:10:001
1839899129,2cyclictest38150irq/131-snd_hda21:25:431
1839899129,2cyclictest27650-21tr22:20:151
1839899129,2cyclictest22800-21grep20:35:171
1839899129,2cyclictest21159-21tail20:30:271
1839899129,2cyclictest17705-21cut23:50:171
1839899129,2cyclictest15158-21ssh22:50:271
1839899129,2cyclictest15158-21ssh22:50:271
1839899129,2cyclictest12435-21grep20:15:151
1839899129,2cyclictest11544-21tr00:25:151
1839899128,3cyclictest27253-21taskset19:38:361
1839899128,2cyclictest15173-21grep21:15:151
1839899120,2cyclictest0-21swapper/123:13:161
1839899120,1cyclictest0-21swapper/123:23:531
1839899120,11cyclictest1679-21grep19:50:241
1839899120,10cyclictest0-21swapper/100:06:491
1839499129,2cyclictest8119-21tail23:25:170
1839499129,2cyclictest6723-21grep20:00:280
1839499129,2cyclictest3306-21head19:55:180
1839499129,2cyclictest32510-21ssh21:25:160
1839499129,2cyclictest31380-21wc23:00:210
1839499129,2cyclictest30701-21systemctl23:20:000
1839499129,2cyclictest30701-21sed22:40:260
1839499129,2cyclictest30003-21sed20:50:170
1839499129,2cyclictest20549-21uniq00:30:180
1839499129,2cyclictest12928-21head20:15:170
1839499128,3cyclictest8081-21grep20:05:200
1839499128,3cyclictest6405-21chrt00:21:320
1839499124,4cyclictest452-21systemd-logind00:06:300
1839499123,5cyclictest225-21systemd-journal21:50:000
1839499120,1cyclictest0-21swapper/023:53:320
1840399119,1cyclictest9175-21ssh00:23:423
1840399118,2cyclictest7402-21cut21:10:173
1840399118,2cyclictest30325-21ssh23:38:443
1840399118,2cyclictest27379-21basename19:40:003
1840399118,2cyclictest26394-21taskset19:30:313
1840399118,2cyclictest2393-21ssh23:40:403
1840399118,2cyclictest21460-21ssh22:55:063
1840399118,2cyclictest21160-21chrt19:17:293
1840399118,2cyclictest19747-21tr19:10:193
1840399118,2cyclictest18089-21expr21:55:143
1840399118,2cyclictest17366-21tr20:25:143
1840399118,2cyclictest14809-21ssh23:29:463
1840399118,2cyclictest10484-21expr20:10:163
1840399117,2cyclictest30948-21head20:50:263
1840399112,2cyclictest6140-21diskmemload22:25:253
18403991111,0cyclictest0-21swapper/321:16:183
1840099118,2cyclictest7421-21cut20:05:172
1840099118,2cyclictest7004-21cut21:10:152
1840099118,2cyclictest5803-21chrt23:05:082
1840099118,2cyclictest31346-21ssh23:58:322
1840099118,2cyclictest30271-21cat23:00:142
1840099118,2cyclictest2990-21python21:00:262
1840099118,2cyclictest29351-21grep20:50:142
1840099118,2cyclictest27319-21sh21:21:522
1840099118,2cyclictest25697-21sed20:40:222
1840099118,2cyclictest2355-21gpgconf19:55:012
1840099118,2cyclictest17576-21grep21:35:192
1840099118,2cyclictest16377-21rm23:30:172
1840099118,2cyclictest13251-21grep20:15:202
1840099118,2cyclictest1313-21awk21:00:012
1840099118,2cyclictest11750-21sshd20:13:272
1840099117,3cyclictest28921-21grep00:15:282
1840099117,3cyclictest19884-21basename22:35:002
1840099111,8cyclictest22323-21head22:35:262
1839899118,2cyclictest8738-21ssh21:30:291
1839899118,2cyclictest5861-21taskset21:07:211
1839899118,2cyclictest29496-21tr20:50:151
1839899118,2cyclictest26934-21sed22:00:181
1839899118,2cyclictest26602-21cron20:45:001
1839899118,2cyclictest25488-21ssh23:16:011
1839899118,2cyclictest22522-21ssh22:55:181
1839899117,3cyclictest2608-21grep19:55:141
1839899117,3cyclictest2057-21sed23:40:261
1839899117,3cyclictest1922-21wc21:45:201
1839899114,5cyclictest29772-21cron19:45:001
1839899111,8cyclictest6140-21diskmemload00:35:191
1839899111,8cyclictest6140-21diskmemload00:35:191
1839499118,2cyclictest8913-21ssh21:50:000
1839499118,2cyclictest8068-21grep21:10:220
1839499118,2cyclictest8014-21rm21:30:190
1839499118,2cyclictest696-21grep19:50:150
1839499118,2cyclictest4135-21ssh23:42:020
1839499118,2cyclictest395-21ssh23:20:250
1839499118,2cyclictest28380-21ssh00:15:230
1839499118,2cyclictest27808-21ssh22:39:400
1839499118,2cyclictest24718-21ssh23:35:150
1839499118,2cyclictest22183-21df22:55:160
1839499118,2cyclictest21641-21perf20:35:000
1839499118,2cyclictest16567-21ssh22:51:290
1839499118,2cyclictest16567-21ssh22:51:290
1839499118,2cyclictest15965-21cut23:30:150
1839499118,2cyclictest15496-21expr21:15:160
1839499118,2cyclictest1205-21chrt20:58:520
1839499118,2cyclictest11994-21sh00:25:160
1839499118,2cyclictest10550-21grep20:10:170
1840399108,1cyclictest4539-21ssh22:45:033
1840399108,1cyclictest4322-21expr21:05:153
1840399108,1cyclictest31919-21cat20:55:123
1840399108,1cyclictest30310-21tr19:45:143
1840399108,1cyclictest241482sleep319:27:113
1840399108,1cyclictest23435-21mailstats20:35:233
1840399108,1cyclictest1423-21wc19:50:243
1840399108,1cyclictest12611-21expr20:15:153
1840399107,2cyclictest4969-21chrt20:00:133
1840399107,2cyclictest28255-21sort19:40:123
1840099108,1cyclictest68362sleep220:01:202
1840099108,1cyclictest27050-21taskset19:36:002
1840099108,1cyclictest24087-21taskset19:26:252
1840099108,1cyclictest21187-21taskset19:17:512
1840099108,1cyclictest14811-21expr20:20:142
1840099107,2cyclictest22747-21grep19:20:152
1840099107,2cyclictest19742-21cut20:30:162
1840099106,3cyclictest18631-21apt-get19:10:122
1840099103,3cyclictest225-21systemd-journal21:07:152
1840099100,1cyclictest0-21swapper/219:30:092
1839899108,1cyclictest6591-21sed20:00:271
1839899108,1cyclictest27084-21cat20:45:161
1839899108,1cyclictest19769-21ntpq19:10:201
1839899108,1cyclictest17012-21apt-config20:25:001
1839899108,1cyclictest1627-21cut21:00:151
1839899107,2cyclictest25212-21ssh23:55:071
1839899100,9cyclictest1195-21chrt20:58:451
1839499108,1cyclictest24232-21chrt19:28:160
1839499108,1cyclictest165822sleep020:24:140
1839499107,2cyclictest4215-21expr21:05:150
1839499107,2cyclictest31782-21taskset19:45:310
1839499107,2cyclictest25763-21processes20:40:220
1839499107,2cyclictest24933-21expr19:30:100
1839499107,2cyclictest22553-21cat20:35:150
1839499107,2cyclictest19766-21awk19:10:200
1839499100,9cyclictest18608-21cat20:25:250
184009997,1cyclictest30650-21grep19:45:162
184009997,1cyclictest220722sleep220:35:102
183989997,2cyclictest25959-21fgrep19:30:181
183949987,1cyclictest28271-21tune2fs19:40:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional